Skip to main content
Top
Published in: BMC Medical Research Methodology 1/2019

Open Access 01-12-2019 | Research article

Which design to evaluate complex interventions? Toward a methodological framework through a systematic review

Authors: Laetitia Minary, Justine Trompette, Joëlle Kivits, Linda Cambon, Cyril Tarquinio, François Alla

Published in: BMC Medical Research Methodology | Issue 1/2019

Login to get access

Abstract

Background

Evaluation of complex interventions (CI) is challenging for health researchers and requires innovative approaches. The objective of this work is to present the main methods used to evaluate CI.

Methods

A systematic review of the scientific literature was conducted to identify methods used for the evaluation of CI. We searched MEDLINE via PubMed databases for articles including an evaluation or a pilot study of a complex intervention, published in a ten-year period. Key-words of this research were (“complex intervention*” AND “evaluation”).

Results

Among 445 identified articles, 100 research results or protocols were included. Among them, 5 presented 2 different types of design in the same publication, thus our work included 105 designs. Individual randomized controlled trials (IRCT) represented 21.9% (n = 23) of evaluation designs, randomized clinical trials adaptations 44.8% (n = 47), quasi -experimental designs and cohort study 19.0% (n = 20), realist evaluation 6.7% (n = 7) and other cases studies and other approaches 8.6% (n = 9). A process/mechanisms analysis was included in 80% (n = 84) of these designs.

Conclusion

A range of methods can be used successively or combined at various steps of the evaluation approach. A framework is proposed to situate each of the designs with respect to evaluation questions. The growing interest of researchers in alternative methods and the development of their use must be accompanied by conceptual and methodological research in order to more clearly define their principles of use.
Literature
1.
go back to reference Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ (Clinical Research Ed.). 1996;312(7040):1215–8.CrossRef Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ (Clinical Research Ed.). 1996;312(7040):1215–8.CrossRef
2.
go back to reference Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc Sci Med. 2012;75(12):2299–306.CrossRef Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc Sci Med. 2012;75(12):2299–306.CrossRef
3.
go back to reference Bonell C, Moore G, Warren E, Moore L. Are randomised controlled trials positivist? Reviewing the social science and philosophy literature to assess positivist tendencies of trials of social interventions in public health and health services. Trials. 2018;19(1):238.CrossRef Bonell C, Moore G, Warren E, Moore L. Are randomised controlled trials positivist? Reviewing the social science and philosophy literature to assess positivist tendencies of trials of social interventions in public health and health services. Trials. 2018;19(1):238.CrossRef
4.
go back to reference Bor J. Capitalizing on natural experiments to improve our understanding of population health. Am J Public Health. 2016;106(8):1388–9.CrossRef Bor J. Capitalizing on natural experiments to improve our understanding of population health. Am J Public Health. 2016;106(8):1388–9.CrossRef
5.
go back to reference Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. 2016;11:63.CrossRef Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. 2016;11:63.CrossRef
7.
go back to reference Cambon L, Minary L, Ridde V, Alla F. Un outil pour accompagner la transférabilité des interventions en promotion de la santé : ASTAIRE. Santé Publique. 2015;26(6):783–6.CrossRef Cambon L, Minary L, Ridde V, Alla F. Un outil pour accompagner la transférabilité des interventions en promotion de la santé : ASTAIRE. Santé Publique. 2015;26(6):783–6.CrossRef
8.
go back to reference Campbell D, Stanley J. Experimental and quasi-experimental designs for research. Chicago: (Rand-McNally); 1963. Campbell D, Stanley J. Experimental and quasi-experimental designs for research. Chicago: (Rand-McNally); 1963.
9.
go back to reference Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P. Framework for design and evaluation of complex interventions to improve health. BMJ Br Med J. 2000;321(7262):694–6.CrossRef Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P. Framework for design and evaluation of complex interventions to improve health. BMJ Br Med J. 2000;321(7262):694–6.CrossRef
15.
go back to reference Donner A, Klar N. Design and analysis of cluster randomization trials in health research. New York: Oxford University Press; 2000. Donner A, Klar N. Design and analysis of cluster randomization trials in health research. New York: Oxford University Press; 2000.
17.
go back to reference Grimshaw J, Campbell M, Eccles M, Steen N. Experimental and quasi-experimental designs for evaluating guideline implementation strategies. Fam Pract. 2000;17(Suppl 1):S11–6.CrossRef Grimshaw J, Campbell M, Eccles M, Steen N. Experimental and quasi-experimental designs for evaluating guideline implementation strategies. Fam Pract. 2000;17(Suppl 1):S11–6.CrossRef
20.
go back to reference Hillsdon M. Evaluating environmental interventions through natural experiments. In: Thorogood M, Coombes Y, editors. Evaluating Health Promotion. New York: Oxford University Press; 2010.CrossRef Hillsdon M. Evaluating environmental interventions through natural experiments. In: Thorogood M, Coombes Y, editors. Evaluating Health Promotion. New York: Oxford University Press; 2010.CrossRef
23.
go back to reference Minary L, Cambon L, Martini H, Wirth N, Acouetey DS, Thouvenot F, et al. Efficacy of a smoking cessation program in a population of adolescent smokers in vocational schools: a public health evaluative controlled study. BMC Public Health. 2013;13(149). https://doi.org/10.1186/1471-2458-13-149. Minary L, Cambon L, Martini H, Wirth N, Acouetey DS, Thouvenot F, et al. Efficacy of a smoking cessation program in a population of adolescent smokers in vocational schools: a public health evaluative controlled study. BMC Public Health. 2013;13(149). https://​doi.​org/​10.​1186/​1471-2458-13-149.
25.
go back to reference Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ (Clinical Research Ed.). 2015b;350:h1258. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ (Clinical Research Ed.). 2015b;350:h1258.
26.
go back to reference Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ (Clinical Research Ed). 2015a;350:h1258. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ (Clinical Research Ed). 2015a;350:h1258.
27.
go back to reference Moore GF, Evans RE. What theory, for whom and in which context? Reflections on the application of theory in the development and evaluation of complex population health interventions. SSM Popul Health. 2017 Dec;3:132–5.CrossRef Moore GF, Evans RE. What theory, for whom and in which context? Reflections on the application of theory in the development and evaluation of complex population health interventions. SSM Popul Health. 2017 Dec;3:132–5.CrossRef
31.
go back to reference Pawson R, Tilley N. Realist evaluation. London: Sage; 1997. Pawson R, Tilley N. Realist evaluation. London: Sage; 1997.
34.
go back to reference Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, Wahlster P, Polus S, Burns J, Brereton L, Rehfuess E. Making sense of complexity in context and implementation: the context and implementation of complex interventions (CICI) framework. Implement Sci. 2017 Feb 15;12(1):21.CrossRef Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, Wahlster P, Polus S, Burns J, Brereton L, Rehfuess E. Making sense of complexity in context and implementation: the context and implementation of complex interventions (CICI) framework. Implement Sci. 2017 Feb 15;12(1):21.CrossRef
35.
go back to reference Roland M, Torgerson DJ. What are pragmatic trials? BMJ (Clinical Research Ed.). 1998;316(7127):285.CrossRef Roland M, Torgerson DJ. What are pragmatic trials? BMJ (Clinical Research Ed.). 1998;316(7127):285.CrossRef
36.
go back to reference Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20(8):637–48.CrossRef Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20(8):637–48.CrossRef
41.
go back to reference Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health. 2004;94(3):400–5.CrossRef Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health. 2004;94(3):400–5.CrossRef
42.
go back to reference Wang S, Moss JR, Hiller JE. Applicability and transferability of interventions in evidence-based public health. Health Promot Int. 2006;21:76–83.CrossRef Wang S, Moss JR, Hiller JE. Applicability and transferability of interventions in evidence-based public health. Health Promot Int. 2006;21:76–83.CrossRef
43.
go back to reference Ward S, Donovan HS, Serlin RC. An alternative view on “an alternative paradigm”. Res Nurs Health. 2003;26(3):256–9.CrossRef Ward S, Donovan HS, Serlin RC. An alternative view on “an alternative paradigm”. Res Nurs Health. 2003;26(3):256–9.CrossRef
44.
go back to reference Weisberg HI, Hayden VC, Pontes VP. Selection criteria and generalizability within the counterfactual framework: explaining the paradox of antidepressant-induced suicidality? Clin Trials. 2009;6(2):109–18.CrossRef Weisberg HI, Hayden VC, Pontes VP. Selection criteria and generalizability within the counterfactual framework: explaining the paradox of antidepressant-induced suicidality? Clin Trials. 2009;6(2):109–18.CrossRef
45.
go back to reference Zelen M. A new design for randomized clinical trials. N Engl J Med. 1979;300(22):1242–5.CrossRef Zelen M. A new design for randomized clinical trials. N Engl J Med. 1979;300(22):1242–5.CrossRef
Metadata
Title
Which design to evaluate complex interventions? Toward a methodological framework through a systematic review
Authors
Laetitia Minary
Justine Trompette
Joëlle Kivits
Linda Cambon
Cyril Tarquinio
François Alla
Publication date
01-12-2019
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2019
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-019-0736-6

Other articles of this Issue 1/2019

BMC Medical Research Methodology 1/2019 Go to the issue