Skip to main content
Top
Published in: BMC Medical Research Methodology 1/2018

Open Access 01-12-2018 | Research article

The generation of consensus guidelines for carrying out process evaluations in rehabilitation research

Authors: P. Masterson-Algar, C. R. Burton, J. Rycroft-Malone

Published in: BMC Medical Research Methodology | Issue 1/2018

Login to get access

Abstract

Background

Although in recent years there has been a strong increase in published research on theories (e.g. realist evaluation, normalization process theory) driving and guiding process evaluations of complex interventions, there is limited guidance to help rehabilitation researchers design and carry out process evaluations. This can lead to the risk of process evaluations being unsystematic. This paper reports on the development of new consensus guidelines that address the specific challenges of conducting process evaluations alongside clinical trials of rehabilitation interventions.

Methods

A formal consensus process was carried out based on a modified nominal group technique, which comprised two phases. Phase I was informed by the findings of a systematic review, and included a nominal group meeting with an expert panel of participants to rate and discuss the proposed statements. Phase II was an in depth semi-structured telephone interviews with expert panel participants in order to further discuss the structure and contents of the revised guidelines. Frequency of rating responses to each statement was calculated and thematic analysis was carried out on all qualitative data.

Results

The guidelines for carrying out process evaluations within complex intervention rehabilitation research were produced by combining findings from Phase I and Phase II. The consensus guidelines include recommendations that are grouped in seven sections. These sections are theoretical work, design and methods, context, recruitment and retention, intervention staff, delivery of the intervention and results. These sections represent different aspects or stages of the evaluation process.

Conclusion

The consensus guidelines here presented can play a role at assisting rehabilitation researchers at the time of designing and conducting process evaluations alongside trials of complex interventions. The guidelines break new ground in terms of concepts and theory and works towards a consensus in regards to how rehabilitation researchers should go about carrying out process evaluations and how this evaluation should be linked into the proposed trials. These guidelines may be used, adapted and tested by rehabilitation researchers depending on the research stage or study design (e.g. feasibility trial, pilot trial, etc.).
Appendix
Available only for authorised users
Literature
1.
go back to reference Robinson L, Francis J, James P, Tindle N, Greenwell K, Rodgers H. Caring for carers of people with stroke: developing a complex intervention following the Medical Research Council framework. Clin Rehabil. 2005;19:560–71.CrossRef Robinson L, Francis J, James P, Tindle N, Greenwell K, Rodgers H. Caring for carers of people with stroke: developing a complex intervention following the Medical Research Council framework. Clin Rehabil. 2005;19:560–71.CrossRef
2.
go back to reference Redfern J, McKevitt C, Wolfe CDA. Development of complex interventions in stroke care: a systematic review. Stroke. 2006;37:2410–9.CrossRef Redfern J, McKevitt C, Wolfe CDA. Development of complex interventions in stroke care: a systematic review. Stroke. 2006;37:2410–9.CrossRef
3.
go back to reference Medical Research Council. Developing and evaluation complex interventions: new guidance. London: Medical Research Council; 2008. Medical Research Council. Developing and evaluation complex interventions: new guidance. London: Medical Research Council; 2008.
4.
go back to reference Horner S, Rew L, Torres R. Enhancing intervention fidelity: a means of strengthening study impact. J Spec Pediatr Nurs. 2006;11(2):80–9.CrossRef Horner S, Rew L, Torres R. Enhancing intervention fidelity: a means of strengthening study impact. J Spec Pediatr Nurs. 2006;11(2):80–9.CrossRef
5.
go back to reference Santacroce SJ, Maccarelli LM, Grey M. Intervention fidelity. Nurse Res. 2004;53(1):63–6.CrossRef Santacroce SJ, Maccarelli LM, Grey M. Intervention fidelity. Nurse Res. 2004;53(1):63–6.CrossRef
6.
go back to reference Kerns SEU, Prinz RJ. Critical issues in the prevention of violence-related behavior in youth. Clin Child Fam Psychol Rev. 2002;5:133–60.CrossRef Kerns SEU, Prinz RJ. Critical issues in the prevention of violence-related behavior in youth. Clin Child Fam Psychol Rev. 2002;5:133–60.CrossRef
7.
go back to reference Hart T, Bagiella E. Design and implementation of clinical trials in rehabilitation research. Arch Physic Med Rehab. 2012;93:S117–26.CrossRef Hart T, Bagiella E. Design and implementation of clinical trials in rehabilitation research. Arch Physic Med Rehab. 2012;93:S117–26.CrossRef
8.
go back to reference Townsend E. Enabling occupation: an occupational therapy perspective. Ottawa: Canadian Association of Occupational Therapists; 2002. Townsend E. Enabling occupation: an occupational therapy perspective. Ottawa: Canadian Association of Occupational Therapists; 2002.
9.
go back to reference Dejong G, Horn SD, Gassaway JA, Slavin MD, Dijkers MP. Toward a taxonomy of rehabilitation interventions: using an inductive approach to examine the “black box” of rehabilitation. Arch Physic Med Rehab. 2004;85:678–86.CrossRef Dejong G, Horn SD, Gassaway JA, Slavin MD, Dijkers MP. Toward a taxonomy of rehabilitation interventions: using an inductive approach to examine the “black box” of rehabilitation. Arch Physic Med Rehab. 2004;85:678–86.CrossRef
11.
go back to reference Richards DA. The complex interventions framework. In: Richards DA, Hallberg IR, editors. Complex interventions in health. An overview of research methods. Abingdon, Oxon: Routledge; 2015. p. 1–15.CrossRef Richards DA. The complex interventions framework. In: Richards DA, Hallberg IR, editors. Complex interventions in health. An overview of research methods. Abingdon, Oxon: Routledge; 2015. p. 1–15.CrossRef
12.
go back to reference Masterson-Algar P, Burton CR, Rycroft-Malone J, Sackley C, Walker M. Towards a programme theory for fidelity in the evaluation of complex interventions. J. Eval. Clin. Pract. 2014;20(4):445–52.CrossRef Masterson-Algar P, Burton CR, Rycroft-Malone J, Sackley C, Walker M. Towards a programme theory for fidelity in the evaluation of complex interventions. J. Eval. Clin. Pract. 2014;20(4):445–52.CrossRef
13.
go back to reference Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. 2013;14:15–25.CrossRef Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. 2013;14:15–25.CrossRef
14.
go back to reference Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation in complex public health intervention studies: the need for guidance. J Epidemiol Community. 2014;68(6):585–6.CrossRef Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation in complex public health intervention studies: the need for guidance. J Epidemiol Community. 2014;68(6):585–6.CrossRef
15.
go back to reference Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. London: MRC Population Health Science Research Network; 2014. Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. London: MRC Population Health Science Research Network; 2014.
16.
go back to reference Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.CrossRef Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.CrossRef
17.
go back to reference Graham I, Logan J, Harrison M, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? The Journal of Continuing Education in the Health Professions. 2006;26:13–24. Graham I, Logan J, Harrison M, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? The Journal of Continuing Education in the Health Professions. 2006;26:13–24.
18.
go back to reference Harrison M, Legare F, Graham I, Fervers B. Adapting clinical practice guidelines to local context and assessing barriers to their use. CMAJ. 2010;182(2):78–84.CrossRef Harrison M, Legare F, Graham I, Fervers B. Adapting clinical practice guidelines to local context and assessing barriers to their use. CMAJ. 2010;182(2):78–84.CrossRef
19.
go back to reference Rycroft-Malone J. Formal consensus: the development of a national clinical guideline. Qual Health Care. 2001;10:238–44.CrossRef Rycroft-Malone J. Formal consensus: the development of a national clinical guideline. Qual Health Care. 2001;10:238–44.CrossRef
20.
go back to reference Grimshaw J, Freemantle N, Wallace S, et al. Developing and implementing clinical practice guidelines. Qual Health Care. 1995;4:55–64.CrossRef Grimshaw J, Freemantle N, Wallace S, et al. Developing and implementing clinical practice guidelines. Qual Health Care. 1995;4:55–64.CrossRef
21.
go back to reference Grimshaw JM, Russell IT. Achieving health gains through clinical guidelines: I. developing scientifically valid guidelines. Qual Assurance Health Care. 1993;2:243–8.CrossRef Grimshaw JM, Russell IT. Achieving health gains through clinical guidelines: I. developing scientifically valid guidelines. Qual Assurance Health Care. 1993;2:243–8.CrossRef
22.
go back to reference Francke AL, Smit MC, de Veer AJ, Mistiaen P. Factors influencing implementation of clinical guidelines for healthcare professionals: a systematic meta-review. BMD Med. Inform. Decis. Mak. 2008;8:38–49.CrossRef Francke AL, Smit MC, de Veer AJ, Mistiaen P. Factors influencing implementation of clinical guidelines for healthcare professionals: a systematic meta-review. BMD Med. Inform. Decis. Mak. 2008;8:38–49.CrossRef
24.
go back to reference Potter M, Gordon S, Hamer P. The nominal group technique: a useful consensus methodology in physiotherapy research. N Z J Physiother. 2004;32(3):126–30. Potter M, Gordon S, Hamer P. The nominal group technique: a useful consensus methodology in physiotherapy research. N Z J Physiother. 2004;32(3):126–30.
25.
go back to reference Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311:376–80.CrossRef Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311:376–80.CrossRef
26.
go back to reference Bloor M, Frankland J, Thomas M, Robson K. Focus groups in social research. London: Sage Publications Ltd; 2001.CrossRef Bloor M, Frankland J, Thomas M, Robson K. Focus groups in social research. London: Sage Publications Ltd; 2001.CrossRef
28.
go back to reference Trickey H, Harvey I, Wilcox G. Formal consensus and consultation: a qualitative method for development of a guideline for dementia. Qual Health Care. 1998;7:192–9.CrossRef Trickey H, Harvey I, Wilcox G. Formal consensus and consultation: a qualitative method for development of a guideline for dementia. Qual Health Care. 1998;7:192–9.CrossRef
29.
go back to reference Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.CrossRef Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.CrossRef
30.
go back to reference Byng R, Norman I, Redfern S, Jones R. Exposing the key functions of a complex intervention for shared care in mental health: case study of a process evaluation. BMC Health Serv Res. 2008;8:274–84.CrossRef Byng R, Norman I, Redfern S, Jones R. Exposing the key functions of a complex intervention for shared care in mental health: case study of a process evaluation. BMC Health Serv Res. 2008;8:274–84.CrossRef
32.
go back to reference Masterson-Algar P, Burton C, Brady MC, Nicoll A, Clarke CE, Rick C, et al. Protocol for the process evaluation of the PD COMM trial: assessing the effectiveness of two types of SLT for people with Parkinson’s disease. Trials. 2017;18:397–405.CrossRef Masterson-Algar P, Burton C, Brady MC, Nicoll A, Clarke CE, Rick C, et al. Protocol for the process evaluation of the PD COMM trial: assessing the effectiveness of two types of SLT for people with Parkinson’s disease. Trials. 2017;18:397–405.CrossRef
33.
go back to reference Clarke DJ, Godfrey M, Hawkins R, Sadler E, et al. Implementing a training intervention to support caregivers after stroke: a process evaluation examining the initiation and embedding of programme change. Implement Sci. 2013. https://doi.org/10.1186/1748-5908-8-96. Clarke DJ, Godfrey M, Hawkins R, Sadler E, et al. Implementing a training intervention to support caregivers after stroke: a process evaluation examining the initiation and embedding of programme change. Implement Sci. 2013. https://​doi.​org/​10.​1186/​1748-5908-8-96.
34.
go back to reference Scianni A, Teixeira-Salmela LF, Ada L. Challenges in recruitment, attendance and adherence of acute stroke survivors to a randomized trial in Brazil: a feasibility study. Rev Bras Fisioter. 2012;16:40–455.CrossRef Scianni A, Teixeira-Salmela LF, Ada L. Challenges in recruitment, attendance and adherence of acute stroke survivors to a randomized trial in Brazil: a feasibility study. Rev Bras Fisioter. 2012;16:40–455.CrossRef
35.
go back to reference Chung JC. An intergenerational reminiscence programme for older adults with early dementia and youth volunteers: values and challenges. Scand J Caring Sci. 2009;23:259–64.CrossRef Chung JC. An intergenerational reminiscence programme for older adults with early dementia and youth volunteers: values and challenges. Scand J Caring Sci. 2009;23:259–64.CrossRef
36.
go back to reference Mayo NE, MacKay-Lyons MJ, Scott SC, et al. A randomized trial of two home-based exercise programmes to improve functional walking post-stroke. Clin Rehabil. 2013;27:659–71.CrossRef Mayo NE, MacKay-Lyons MJ, Scott SC, et al. A randomized trial of two home-based exercise programmes to improve functional walking post-stroke. Clin Rehabil. 2013;27:659–71.CrossRef
37.
go back to reference Letts L, Dunal L. Tackling evaluation: applying a programme logic model to community rehabilitation for adults with brain injury. Can J Occup Ther. 1995;62:268–77.CrossRef Letts L, Dunal L. Tackling evaluation: applying a programme logic model to community rehabilitation for adults with brain injury. Can J Occup Ther. 1995;62:268–77.CrossRef
38.
go back to reference Cohn S, Clinch M, Bunn C, Stronge P. Entangled complexity: why complex interventions are just not complicated enough. J Health Serv Res Policy. 2013;18(1):40–3.CrossRef Cohn S, Clinch M, Bunn C, Stronge P. Entangled complexity: why complex interventions are just not complicated enough. J Health Serv Res Policy. 2013;18(1):40–3.CrossRef
39.
go back to reference Wolf S, Winstein C, Miller JP, Thompson PA, Taub E, Uswatte G, Morris D, Blanton S, Nichols-Larsen D. The EXCITE trial: retention of improved upper extremity function among stroke survivors receiving CI movement therapy. Lancet Neurol. 2008;7(1):33–40.CrossRef Wolf S, Winstein C, Miller JP, Thompson PA, Taub E, Uswatte G, Morris D, Blanton S, Nichols-Larsen D. The EXCITE trial: retention of improved upper extremity function among stroke survivors receiving CI movement therapy. Lancet Neurol. 2008;7(1):33–40.CrossRef
40.
go back to reference Schulz KF, Altman DG, Moher D, CONSORT Group, et al. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. BMJ. 2010;340:c332.CrossRef Schulz KF, Altman DG, Moher D, CONSORT Group, et al. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. BMJ. 2010;340:c332.CrossRef
41.
go back to reference Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.CrossRef Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.CrossRef
42.
go back to reference Oakley A, Strange V, Bonell C, Allen E, Stephenson J. Process evaluation in randomised controlled trials of complex interventions. BMJ Health Serv Res. 2006;332:413–6. Oakley A, Strange V, Bonell C, Allen E, Stephenson J. Process evaluation in randomised controlled trials of complex interventions. BMJ Health Serv Res. 2006;332:413–6.
43.
go back to reference Pawson R, Tilley N. Realistic Evaluation. London: Sage Publications; 1997. Pawson R, Tilley N. Realistic Evaluation. London: Sage Publications; 1997.
44.
go back to reference Morrison DDM. Replicating an intervention: the tension between fidelity and adaptation. AIDS Educ Prev. 2009;21(2):128–40.CrossRef Morrison DDM. Replicating an intervention: the tension between fidelity and adaptation. AIDS Educ Prev. 2009;21(2):128–40.CrossRef
45.
go back to reference Song M, Happ MB, Sandelowski M. Development of a tool to assess fidelity to a psycho-educational intervention. J Adv Nurs. 2010;66(3):673–82.CrossRef Song M, Happ MB, Sandelowski M. Development of a tool to assess fidelity to a psycho-educational intervention. J Adv Nurs. 2010;66(3):673–82.CrossRef
46.
go back to reference Elliot D, Mihalic S. Issues in disseminating and replicating effective prevention programs. Prev. Sci. 2004;5:47–53.CrossRef Elliot D, Mihalic S. Issues in disseminating and replicating effective prevention programs. Prev. Sci. 2004;5:47–53.CrossRef
47.
go back to reference Mihalic S. The importance of implementation fidelity. Emot & Behav Disord in Youth. 2004;4:83–105. Mihalic S. The importance of implementation fidelity. Emot & Behav Disord in Youth. 2004;4:83–105.
48.
go back to reference O’Cathain A, Murphy E, Nicholl J. Multidisciplinary, interdisciplinary, or dysfunctional? Team working in mixed-methods research. Qual Health Res. 2008;18:1574–85.CrossRef O’Cathain A, Murphy E, Nicholl J. Multidisciplinary, interdisciplinary, or dysfunctional? Team working in mixed-methods research. Qual Health Res. 2008;18:1574–85.CrossRef
49.
go back to reference Audrey S, Holliday J, Parry-Langdon N, Campbell R. Meeting the challenges of implementing process evaluation within randomized controlled trials: the example of ASSIST (a stop smoking in schools trial). Health Educ Res. 2006;21:366–77.CrossRef Audrey S, Holliday J, Parry-Langdon N, Campbell R. Meeting the challenges of implementing process evaluation within randomized controlled trials: the example of ASSIST (a stop smoking in schools trial). Health Educ Res. 2006;21:366–77.CrossRef
Metadata
Title
The generation of consensus guidelines for carrying out process evaluations in rehabilitation research
Authors
P. Masterson-Algar
C. R. Burton
J. Rycroft-Malone
Publication date
01-12-2018
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2018
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-018-0647-y

Other articles of this Issue 1/2018

BMC Medical Research Methodology 1/2018 Go to the issue