Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
BMC Public Health
Published in: BMC Public Health 1/2013

Open Access 01-12-2013 | Research article

Challenges to evaluating complex interventions: a content analysis of published papers

Authors: Jessica Datta, Mark Petticrew

Published in: BMC Public Health | Issue 1/2013

Login to get access

Abstract

Background

There is continuing interest among practitioners, policymakers and researchers in the evaluation of complex interventions stemming from the need to further develop the evidence base on the effectiveness of healthcare and public health interventions, and an awareness that evaluation becomes more challenging if interventions are complex.
We undertook an analysis of published journal articles in order to identify aspects of complexity described by writers, the fields in which complex interventions are being evaluated and the challenges experienced in design, implementation and evaluation. This paper outlines the findings of this documentary analysis.

Methods

The PubMed electronic database was searched for the ten year period, January 2002 to December 2011, using the term “complex intervention*” in the title and/or abstract of a paper. We extracted text from papers to a table and carried out a thematic analysis to identify authors’ descriptions of challenges faced in developing, implementing and evaluating complex interventions.

Results

The search resulted in a sample of 221 papers of which full text of 216 was obtained and 207 were included in the analysis. The 207 papers broadly cover clinical, public health and methodological topics. Challenges described included the content and standardisation of interventions, the impact of the people involved (staff and patients), the organisational context of implementation, the development of outcome measures, and evaluation.

Conclusions

Our analysis of these papers suggests that more detailed reporting of information on outcomes, context and intervention is required for complex interventions. Future revisions to reporting guidelines for both primary and secondary research may need to take aspects of complexity into account to enhance their value to both researchers and users of research.
Literature
1.
Craig P, Dieppe P, Macintyre S, Ritchie S, Nazareth I, Petticrew M: Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008, 337: a1655-10.1136/bmj.a1655.CrossRefPubMedPubMedCentral
2.
Treweek S: Complex interventions and the chamber of secrets: understanding why they work and why they do not. J R Soc Med. 2005, 98: 553-10.1258/jrsm.98.12.553.CrossRefPubMedPubMedCentral
3.
Mills A, Gilson L, Hanson K, Palmer N, Lagarde M: What do we mean by rigorous health-systems research?. Lancet. 2008, 372: 1527-1529. 10.1016/S0140-6736(08)61633-5.CrossRefPubMed
4.
Medical Research Council: A framework for development and evaluation of RCTs for complex interventions to improve health. 2000, London: MRC
5.
Medical Research Council: Developing and evaluating complex interventions: new guidance. 2008, London: MRC
6.
Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. Edited by: Higgins JPT, Green S. 2011, London: The Cochrane Collaboration
7.
Sheill A, Hawe P, Gold L: Complex interventions or complex systems? Implications for health economic evaluation. BMJ. 2008, 336: 1281-10.1136/bmj.39569.510521.AD.CrossRef
8.
Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P: Framework for design and evaluation of complex interventions to improve health. BMJ. 2000, 321: 694-10.1136/bmj.321.7262.694.CrossRefPubMedPubMedCentral
9.
Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, Guthrie B, Lester H, Wilson P, Kinmonth AL: Designing and evaluating complex interventions to improve health care. BMJ. 2007, 334: 455-10.1136/bmj.39108.379965.BE.CrossRefPubMedPubMedCentral
10.
Hawe P, Shiell A, Riley T: Complex interventions: how “out of control” can a randomised controlled trial be?. BMJ. 2004, 328: 1561-10.1136/bmj.328.7455.1561.CrossRefPubMedPubMedCentral
11.
Oakley A, Strange V, Bonell C, Allen E, Stephenson J: Process evaluation in randomised control trials of complex interventions. BMJ. 2006, 332: 413-10.1136/bmj.332.7538.413.CrossRefPubMedPubMedCentral
12.
Price S, Long AF, Godfrey M, Thomas KJ: Getting inside acupuncture trials -exploring intervention theory and rationale. BMC Complement Altern Med. 2011, 11: 22-10.1186/1472-6882-11-22.CrossRefPubMedPubMedCentral
13.
Bennell KL, Egerton T, Pua YH J, Abbott JH, Sims K, Buchbinder R: Building the rationale and structure for a complex physical therapy intervention within the context of a clinical trial: a multimodal individualised treatment for patients with hip osteoarthritis. Phys The. 2011, 91: 1-17.
14.
Borglin G, Gustaffson M, Krona H: A theory-based educational intervention targeting nurses’ attitudes and knowledge concerning cancer-related pain management: a study protocol of a quasi-experimental design. BMC Health Serv Res. 2011, 11: 233-10.1186/1472-6963-11-233.CrossRefPubMedPubMedCentral
15.
Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, Finch T, Kennedy A, Mair F, O’Donnell C, Ong BN, Rapley T, Rogers A, May C: Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med. 2010, 8: 63-10.1186/1741-7015-8-63.CrossRefPubMedPubMedCentral
16.
Forster DA, Newton M, McLachlan HL, Willis K: Exploring implementation and sustainability of models of care: can theory help?. BMC Publ Health. 2011, 11 (Suppl 5): S8-10.1186/1471-2458-11-S5-S8.CrossRef
17.
Bower P, Gilbody S, Richards D, Fletcher J, Sutton A: Collaborative care for depression in primary care: making sense of a complex intervention: systematic review and meta-regression. Br J Psychiatry. 2006, 189: 484-493. 10.1192/bjp.bp.106.023655.CrossRefPubMed
18.
Hawe P, Shiell A, Riley T: In response to Spillane V., Byrne M.C., Byrne M., Leathem C.S., O’Malley M. & Cupples M.E. (2007) Monitoring treatment fidelity in a randomized trial of a complex intervention. Journal of Advanced Nursing 60(3), 343–352. J Adv Nurs. 2008, 62: 267-10.1111/j.1365-2648.2008.04686.x.CrossRefPubMed
19.
Wilson PM: The UK Expert Patients Programme: lessons learned and implications for cancer survivors' self-care support programmes. J Cancer Surviv. 2008, 2: 45-52. 10.1007/s11764-007-0040-z.CrossRefPubMed
20.
Bird L, Arthur A, Cox K: ‘Did the trial kill the intervention?’ experiences from the development, implementation and evaluation of a complex intervention. BMC Med Res Methodol. 2011, 11: 24-10.1186/1471-2288-11-24.CrossRefPubMedPubMedCentral
21.
Simon S, Higginson IJ: Evaluation of hospital palliative care teams: strengths and weaknesses of the before-after study design and strategies to improve it. Palliat Med. 2009, 23: 23-28.CrossRefPubMed
22.
Schroer S, Adamson J: Acupuncture for depression: a critique of the evidence base. CNS Neurosci Ther. 2011, 17: 398-410. 10.1111/j.1755-5949.2010.00159.x.CrossRefPubMed
23.
Brady MC, Stott DJ, Norrie J, Chalmers C, St George B, Sweeney PM, Langhorne P: Developing and evaluating the implementation of a complex intervention: using mixed methods to inform the design of a randomised controlled trial of an oral healthcare intervention after stroke. Trials. 2011, 12: 168-10.1186/1745-6215-12-168.CrossRefPubMedPubMedCentral
24.
Turner DE, Helliwell PS, Woodburn J: Methodological considerations for a randomised controlled trial of podiatry care in rheumatoid arthritis: lessons from an exploratory trial. BMC Musculoskelet Disord. 2007, 8: 109-10.1186/1471-2474-8-109.CrossRefPubMedPubMedCentral
25.
Spillane V, Byrne MC, Byrne M, Leathem CS, O’Malley M, Cupples ME: Monitoring treatment fidelity in a randomized controlled trial of a complex intervention. J Adv Nurs. 2007, 60: 343-352. 10.1111/j.1365-2648.2007.04386.x.CrossRefPubMed
26.
Tee A, Calzavacca P, Licari E, Goldsmith D, Bellemo R: Bench-to-bedside review: the MET syndrome–the challenges of researching and adopting medical emergency teams. Crit Care. 2008, 12: 205-10.1186/cc6426.CrossRefPubMedPubMedCentral
27.
Cousins K, Connor JL, Kypri K: Reducing alcohol-related harm and social disorder in a university community: a framework for evaluation. Inj Prev. 2010, 16: e1-CrossRefPubMed
28.
Hoddinott P, Pill R, Chalmers M: Health professionals, implementation and outcomes: reflections on a complex intervention to improve breastfeeding rates in primary care. Fam Prac. 2007, 24: 84-91.CrossRef
29.
Power R, Langhaug L, Cowan F: ‘But there are no snakes in the wood’: risk mapping as an outcome measure in evaluating complex interventions. Sex Transm Infect. 2007, 83: 232-236.CrossRefPubMedPubMedCentral
30.
Allegrante JP, Peterson MGE, Cornell CN, MacKenzie : Methodological challenges of multiple-component intervention: lessons learned from a randomized controlled trial of functional recovery after hip fracture. HSS J. 2007, 3: 63-70. 10.1007/s11420-006-9036-x.CrossRefPubMed
31.
Schiapparelli P, Allais G, Rolando S, Airola G, Borgogno P, Terzi MG, Benedetto C: Acupuncture in primary headache treatment. Neurol Sci. 2011, 32 (Suppl 1): S15-S18.CrossRefPubMed
32.
Farquhar M, Higginson IJ, Fagan P, Booth S: Results of a pilot investigation into a complex intervention for breathlessness in advanced chronic obstructive pulmonary disease (COPD): brief report. Palliat Support Care. 2010, 8: 143-149. 10.1017/S1478951509990897.CrossRefPubMed
33.
Dunn G, Fowler D, Rollinson R, Freeman D, Kuipers E, Smith B, Steel C, Onwumere J, Jolley S, Garety P, Bebbington P: Effective elements of cognitive behaviour therapy for psychosis: results of a novel type of subgroup analysis based on principal stratification. Psychol Med. 2011, 42: 1057-1068.CrossRefPubMedPubMedCentral
34.
Gask L, Dowrick C, Salmon P, Peters S, Morriss R: Reattribution reconsidered: narrative review and reflections on an educational intervention for medically unexplained symptoms in primary care settings. J Psychosom Res. 2011, 71: 325-334. 10.1016/j.jpsychores.2011.05.008.CrossRefPubMed
35.
Sampson EL, Jones L, Thuné-Boyle ICV, Kukkastenvehmas R, King M, Leurent B, Tookman A, Blanchard MR: Palliative assessment and advance care planning in severe dementia: an exploratory randomized controlled trial of a complex intervention. Palliat Med. 2011, 25: 197-209. 10.1177/0269216310391691.CrossRefPubMed
36.
Garfein RS, Swartzendruber A, Ouellet LJ, Kapadia F, Hudson SM, Thiede H, Strathdee SA, Williams IT, Bailey SL, Hagan H, Golub ET, Kerndt P, Hanson DL, Latka MH for the DUIT Study Team: Methods to recruit and retain a cohort of young-adult injection drug users for the Third Collaborative Injection Drug Users Study/Drug Users Intervention Trial (CIDUS III/DUIT). Drug Alcohol Depend. 2007, 91 (Suppl 1): S4-S17.CrossRefPubMed
37.
Lyles JS, Hodges A, Collins C, Lein C, Given CW, Given B, D’Mello D, Osborn GG, Goddeeris J, Gardiner JC, Smith RC: Using nurse practitioners to implement an intervention in primary care for high-utilizing patients with medically unexplained symptoms. Gen Hosp Psychiatry. 2003, 25: 63-73. 10.1016/S0163-8343(02)00288-8.CrossRefPubMed
38.
Van Herck P, Vanhaecht K, Deneckere S, Bellemans J, Panella , Barbieri A, Sermeus W: Key interventions and outcomes in joint arthroplasty clinical pathways: a systematic review. J Eval Clin Pract. 2010, 16: 39-49. 10.1111/j.1365-2753.2008.01111.x.CrossRefPubMed
39.
Gustafsson L, Nugent N, Biron L: Occupational therapy practice in hospital-based stroke rehabilitation?. Scand J Occup Ther. 2012, 19: 132-139. 10.3109/11038128.2011.562915.CrossRefPubMed
40.
Hoddinott P, Britten J, Pill R: Why do interventions work in some places and not others: a breastfeeding support group trial. Soc Sci Med. 2010, 70: 769-778. 10.1016/j.socscimed.2009.10.067.CrossRefPubMed
41.
Salas I: Methodology for reorganization of the cervical cancer program in Chile. Cancer Detect Prev. 2006, 30: 38-43. 10.1016/j.cdp.2005.11.003.CrossRefPubMed
42.
Byng R, Norman I, Refern S, Jones R: Exposing the key functions of a complex intervention for shared care in mental health: case study of a process evaluation. BMC Health Serv Res. 2008, 8: 274-10.1186/1472-6963-8-274.CrossRefPubMedPubMedCentral
43.
Mayo NE, Scott S: Evaluating a complex intervention with a single outcome may not be a good idea: an example from a randomised trial of stroke case management. Age Ageing. 2011, 40: 718-724. 10.1093/ageing/afr061.CrossRefPubMed
44.
de Vlaming R, Haveman-Nies A, de Groot LCPGM, van’t Veer P: Evaluation design for a complex intervention program targeting loneliness in non-institutionalized elderly Dutch people. BMC Publ Health. 2010, 10: 552-10.1186/1471-2458-10-552.CrossRef
45.
Klinkhammer-Schalke M, Koller M, Ehret C, Steinger B, Ernst B, Wyatt JC, Hofstädter F, Lorenz W, for the Regensburg QoL Study Group: Implementing a system of quality-of-life diagnosis and therapy for breast cancer patients: results of an exploratory trial as a prerequisite for a subsequent RCT. Br J Cancer. 2008, 99: 415-422. 10.1038/sj.bjc.6604505.CrossRefPubMedPubMedCentral
46.
Farquhar MC, Ewing G, Booth S: Using mixed methods to develop and evaluate complex interventions in palliative care research. Palliat Med. 2011, 25: 748-757. 10.1177/0269216311417919.CrossRefPubMed
47.
Paterson C, Britten N: Acupuncture as a complex intervention: a holistic model. J Altern Complement Med. 2004, 10: 791-801.CrossRefPubMed
48.
Rowlands G, Sims J, Kerry S: A lesson learnt: the importance of modelling in randomized controlled trials for complex interventions in primary care. Fam Pract. 2005, 22: 132-139.CrossRefPubMed
49.
Hirsch O, Keller H, Krones T, Donner-Banzhoff N: Acceptance of shared decision making with reference to an electronic library of decision aids (arriba-lib) and its association to decision making in patients: an evaluation study. Implement Sci. 2011, 6: 70-10.1186/1748-5908-6-70.CrossRefPubMedPubMedCentral
50.
Edwards ED, Mason BW, Oliver A, Powell CVE: Cohort study to test the predictability of the Melbourne criteria for activation of the medical emergency team. Arch Dis Child. 2011, 96: 174-179. 10.1136/adc.2010.187617.CrossRefPubMed
51.
Drescher U, Warren F, Norton K: Towards evidence-based practice in medical training: making evaluations more meaningful. Med Educ. 2004, 38: 1288-1294. 10.1111/j.1365-2929.2004.02021.x.CrossRefPubMed
52.
Pope C, Mays N: Qualitative research: reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ. 1995, 311: 42-10.1136/bmj.311.6996.42.CrossRefPubMedPubMedCentral
53.
Donovan JL, Lane JA, Peters TJ, Brindle L, Salter E, Gillatt D, Powell P, Bollina P, Neal DE, Hamdy FC: Development of a complex intervention improved randomisation and informed consent in a randomised controlled trial. J Clin Epidemiol. 2009, 62: 29-36. 10.1016/j.jclinepi.2008.02.010.CrossRefPubMed
54.
Anderson L, Petticrew M, Rehfuess E, Armstrong R, Ueffing E, Baker P, Francis D, Tugwell P: Using logic models to capture complexity in systematic reviews. Res Synth Methods. 2011, 2: 33-42. 10.1002/jrsm.32.CrossRefPubMed
Metadata
Title
Challenges to evaluating complex interventions: a content analysis of published papers
Authors
Jessica Datta
Mark Petticrew
Publication date
01-12-2013
Publisher
BioMed Central
Published in
BMC Public Health / Issue 1/2013
Electronic ISSN: 1471-2458
DOI
https://doi.org/10.1186/1471-2458-13-568

Other articles of this Issue 1/2013

BMC Public Health 1/2013 Go to the issue

Research article

Explaining the decline in coronary heart disease mortality in Turkey between 1995 and 2008

Research article

Outdoor fitness equipment in parks: a qualitative study from older adults’ perceptions

Research article

Exploring the potential for using results-based financing to address non-communicable diseases in low- and middle-income countries

Research article

Developing social capital in implementing a complex intervention: a process evaluation of the early implementation of a suicide prevention intervention in four European countries

Research article

Change in quality of life: a follow up study among patients with HIV infection with and without TB in Ethiopia

Research article

Clustering of risk factors for non-communicable disease and healthcare expenditure in employees with private health insurance presenting for health risk appraisal: a cross-sectional study