Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
Systematic Reviews
Published in: Systematic Reviews 1/2017

Open Access 01-12-2017 | Methodology

Selecting and implementing overview methods: implications from five exemplar overviews

Authors: Alex Pollock, Pauline Campbell, Ginny Brunton, Harriet Hunt, Lise Estcourt

Published in: Systematic Reviews | Issue 1/2017

Login to get access

Abstract

Background

Overviews of systematic reviews are an increasingly popular method of evidence synthesis; there is a lack of clear guidance for completing overviews and a number of methodological challenges. At the UK Cochrane Symposium 2016, methodological challenges of five overviews were explored. Using data from these five overviews, practical implications to support methodological decision making of authors writing protocols for future overviews are proposed.

Methods

Methods, and their justification, from the five exemplar overviews were tabulated and compared with areas of debate identified within current literature. Key methodological challenges and implications for development of overview protocols were generated and synthesised into a list, discussed and refined until there was consensus.

Results

Methodological features of three Cochrane overviews, one overview of diagnostic test accuracy and one mixed methods overview have been summarised. Methods of selection of reviews and data extraction were similar. Either the AMSTAR or ROBIS tool was used to assess quality of included reviews. The GRADE approach was most commonly used to assess quality of evidence within the reviews.
Eight key methodological challenges were identified from the exemplar overviews. There was good agreement between our findings and emerging areas of debate within a recent published synthesis. Implications for development of protocols for future overviews were identified.

Conclusions

Overviews are a relatively new methodological innovation, and there are currently substantial variations in the methodological approaches used within different overviews. There are considerable methodological challenges for which optimal solutions are not necessarily yet known. Lessons learnt from five exemplar overviews highlight a number of methodological decisions which may be beneficial to consider during the development of an overview protocol.
Appendix
Available only for authorised users
Literature
1.
Hartling L, Chisholm A, Thomson D, Dryden DM. A descriptive analysis of overviews of reviews published between 2000 and 2011. PLoS ONE. 2012;7, e49667.CrossRefPubMedPubMedCentral
2.
Pieper D, Buechter R, Jerinic P, Eikermann M. Overviews of reviews often have limited rigor: a systematic review. J Clin Epidemiol. 2012;65:1267–73.CrossRefPubMed
3.
Ballard M, Montgomery P. Risk of bias in overviews of reviews: a scoping review of methodological guidance and four-item checklist. Research Synthesis Methods. 2017;8:92–108.CrossRefPubMed
4.
Aromataris E, Fernandez R, Godfrey CM, Holly C, Khalil H, Tungpunkom P. Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. Int J Evid Based Healthc. 2015;13:132–40.CrossRefPubMed
5.
Pollock M, Fernandes R, Becker L, Featherstone R, Hartling L. What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary. Systematic Reviews. 2016;5:190.CrossRefPubMedPubMedCentral
6.
Thomson D, Foisy M, Oleszczuk M, Wingert A, Chisholm A, Hartling L. Overview of reviews in child health: evidence synthesis and the knowledge base for a specific population. Evidence-Based Child Health: A Cochrane Review Journal. 2013;8:3–10.CrossRef
7.
Caird J, Sutcliffe K, Kwan I, Dickson K, Thomas J. Mediating policy-relevant evidence at speed: Are systematic reviews of systematic reviews a useful approach? Evidence and Policy. 2015;11:81-97.
8.
Smith V, Devane D, Begley CM, Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol. 2011;11:15.CrossRefPubMedPubMedCentral
9.
Pollock A, Hunt H, Campbell P, Estcourt L, Brunton G. Cochrane overviews of reviews: exploring the methods and challenges. Birmingham: UK and Ireland Cochrane Symposium; 2016.
10.
Pollock A, Farmer SE, Brady MC, Langhorne P, Mead GE, Mehrholz J, Van Wijck F, Wiffen PJ. An algorithm was developed to assign GRADE levels of evidence to comparisons within systematic reviews. J Clin Epidemiol. 2016;70:106–10.CrossRefPubMedPubMedCentral
11.
Murad MH, Mustafa R, Morgan R, Sultan S, Falck-Ytter Y, Dahm P. Rating the quality of evidence is by necessity a matter of judgment. J Clin Epidemiol. 2016;74:237–8.CrossRefPubMed
12.
Gionfriddo MR. Subjectivity is a strength: a comment on “an algorithm was developed to assign GRADE levels of evidence to comparisons within systematic reviews”. J Clin Epidemiol. 2016;74:237.CrossRefPubMed
13.
Pollock M, Fernandes RM, Hartling L. Evaluation of AMSTAR to assess the methodological quality of systematic reviews in overviews of reviews of healthcare interventions. BMC Med Res Methodol. 2017;17:48.CrossRefPubMedPubMedCentral
14.
Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R, group R. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.CrossRefPubMedPubMedCentral
15.
Lunny C, Brennan SE, McDonald S, McKenzie JE. Evidence map of studies evaluating methods for conducting, interpreting and reporting overviews of systematic reviews of interventions: rationale and design. Systematic Reviews. 2016;5:4.CrossRefPubMedPubMedCentral
16.
17.
Erzberger C, Prein G. Triangulation: validity and empirically-based hypothesis construction. Qual Quant. 1997;31:141–54.CrossRef
18.
O’Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. BMJ. 2010;341.
19.
Pollock A, Farmer SE, Brady MC, Langhorne P, Mead GE, Mehrholz J, Van Wijck F. Interventions for improving upper limb function after stroke. Cochrane Database Syst Rev. 2014;11:CD010820.
20.
Estcourt LJ, Fortin PM, Hopewell S, Trivella M: Red blood cell transfusion to treat or prevent complications in sickle cell disease: an overview of Cochrane reviews. Cochrane Database Syst Rev 2016, 2016.
21.
McClurg D, Pollock A, Campbell P, Hazelton C, Elders A, Hagen S, Hill DC, McClurg D: Conservative interventions for urinary incontinence in women: an Overview of Cochrane systematic reviews. Cochrane Database of Systematic Reviews 2016.
22.
Hunt H, Kuzma E, C H: A review of existing systematic reviews summarising the accuracy of brief cognitive assessments for identifying dementia, particularly for use in primary care. Protocol..In PROSPERO PROSPERO online; 2016.
23.
Brunton G, Dickson K, Khatwa M, Caird J, Oliver S, Hinds K, Thomas J. Developing evidence-informed, employer-led workplace health. London: EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London; 2016.
24.
Li L, Tian J, Tian H, Sun R, Liu Y, Yang K. Quality and transparency of overviews of systematic reviews. Journal of Evidence-Based Medicine. 2012;5:166–73.CrossRefPubMed
25.
Pieper D, Antoine S-L, Morfeld J-C, Mathes T, Eikermann M. Methodological approaches in conducting overviews: current state in HTA agencies. Research Synthesis Methods. 2014;5:187–99.CrossRefPubMed
26.
Boote J, Wong R, Booth A. ‘Talking the talk or walking the walk?’ A bibliometric review of the literature on public involvement in health research published between 1995 and 2009. Health Expect. 2015;18:44–57.CrossRefPubMed
27.
INVOLVE. Public involvement in systematic reviews: Supplement to the briefing notes for researchers. Eastleigh: INVOLVE; 2012.
28.
Kreis J, Puhan MA, Schunemann HJ, Dickersin K. Consumer involvement in systematic reviews of comparative effectiveness research. Health Expect. 2013;16:323–37.CrossRefPubMed
29.
Morley R, Norman G, Golder S, Griffith P. A systematic scoping review of the evidence for consumer involvement in organisations undertaking systematic reviews: focus on Cochrane. Research Involvement and Engagement. 2016;2:36.CrossRef
30.
Serrano-Aguilar P, Trujillo-Martin MM, Ramos-Goni JM, Mahtani-Chugani V, Perestelo-Perez L, Posada-dela Paz M. Patient involvement in health research: a contribution to a systematic review on the effectiveness of treatments for degenerative ataxias. Soc Sci Med. 2009;69:920–5.CrossRefPubMed
31.
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896–900.CrossRefPubMed
32.
MECIR. The Methodological Expectations of Cochrane Intervention Reviews (MECIR).Standards for Cochrane new reviews of interventions and their updates. 2017.
33.
Becker L, Oxman A: Chapter 22: Overviews of reviews. In Cochrane Handbook for Systematic Reviews of Interventions Version 510 (JPT H, S G eds.): The Cochrane Collaboration; 2011. Available from www.​handbook.​cochrane.​org.
34.
35.
Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, Goodare H, Watts C, Morley R. Stakeholder involvement in systematic reviews: a protocol for a systematic review of methods, outcomes and effects. Research Involvement and Engagement. 2017;3:9.CrossRef
36.
37.
Pollock A, Brady MC, Farmer SE, Langhorne P, Mead GE, Mehrholz J, Wiffen PJ, Van Wijck F. The purpose of rating quality of evidence differs in an overview, as compared to guidelines or recommendations. J Clin Epidemiol. 2016;74:238–40.CrossRefPubMed
38.
Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Moher D, Rennie D, De Vet HC, Lijmer JG, Standards for Reporting of Diagnostic A. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med. 2003;138:W1–12.CrossRefPubMed
39.
Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM, Group Q. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155:529–36.CrossRefPubMed
40.
PSTF. U.S. Preventive Services Task Force. Procedure Manual. In: AHRQ publication no 08-05118-EF. Rockville: U.S. Preventive Services Task Force; 2008.
41.
NICE. National Institute for Health and Clinical Excellence. The Guidelines Manual. London: National Institute for Health and Clinical Excellence; 2006.
42.
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.CrossRefPubMedPubMedCentral
43.
NOS. Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa: Ottawa Hospital Research Institute; 2013.
44.
Estcourt LJ, Fortin PM, Hopewell S, Trivella M, Hambleton IR, Cho G. Regular long-term red blood cell transfusions for managing chronic chest complications in sickle cell disease. Cochrane Database Syst Rev. 2016;20:CD008360.
45.
Estcourt LJ, Fortin PM, Trivella M, Hopewell S. Preoperative blood transfusions for sickle cell disease. Cochrane Database Syst Rev. 2016;4, CD003149.PubMedPubMedCentral
46.
Estcourt LJ, Fortin PM, Hopewell S, Trivella M, Wang WC. Blood transfusion for preventing primary and secondary stroke in people with sickle cell disease. Cochrane Database Syst Rev. 2017;1, CD003146.PubMed
47.
Roy NB, Fortin PM, Bull KR, Doree C, Trivella M, Hopewell S, Estcourt LJ: Interventions for chronic kidney disease in people with sickle cell disease. In Cochrane Database of Systematic Reviews: John Wiley & Sons, Ltd; 2016.
48.
Estcourt LJ, Fortin PM, Hopewell S, Trivella M, Doree C, Abboud MR: Interventions for preventing silent cerebral infarcts in people with sickle cell disease. In Cochrane Database of Systematic Reviews: John Wiley & Sons, Ltd; 2016.
49.
Okusanya BO, Oladapo OT. Prophylactic versus selective blood transfusion for sickle cell disease in pregnancy. Cochrane Database Syst Rev. 2013;3:CD010378.
50.
Dastgiri S, Dolatkhah R: Blood transfusions for treating acute chest syndrome in people with sickle cell disease. In Cochrane Database of Systematic Reviews: John Wiley & Sons, Ltd; 2016.
51.
Martí-Carvajal AJ, Knight-Madden JM, Martinez-Zapata MJ: Interventions for treating leg ulcers in people with sickle cell disease. In Cochrane Database of Systematic Reviews: John Wiley & Sons, Ltd; 2014.
52.
53.
54.
Pieper D, Antoine SL, Mathes T, Neugebauer EA, Eikermann M. Systematic review finds overlapping reviews were not mentioned in every other overview. J Clin Epidemiol. 2014;67:368–75.CrossRefPubMed
55.
56.
Pieper D, Antoine SL, Neugebauer EA, Eikermann M. Up-to-dateness of reviews is often neglected in overviews: a systematic review. J Clin Epidemiol. 2014;67:1302–8.CrossRefPubMed
57.
58.
59.
Burda BU, Holmer HK, Norris SL. Limitations of A Measurement Tool to Assess Systematic Reviews (AMSTAR) and suggestions for improvement. Systematic Reviews. 2016;5:58.CrossRefPubMedPubMedCentral
60.
Faggion CM. Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor. BMC Med Res Methodol. 2015;15:63.CrossRefPubMedPubMedCentral
61.
Wegewitz U, Weikert B, Fishta A, Jacobs A, Pieper D. Resuming the discussion of AMSTAR: what can (should) be made better? BMC Med Res Methodol. 2016;16:111.CrossRefPubMedPubMedCentral
62.
EPOC. Effective Practice and Organisation of Care (EPOC): reporting the effects of an intervention in EPOC reviews. Section 24. How to report the effects of an intervention. In: EPOC Resources for review authors. Oslo: Norwegian Knowledge Centre for the Health Services; 2016.
63.
Glenton C, Santesso N, Rosenbaum S, Nilsen ES, Rader T, Ciapponi A, Dilkes H. Presenting the results of Cochrane Systematic Reviews to a consumer audience: a qualitative study. Med Decis Making. 2010;30:566–77.CrossRefPubMed
Metadata
Title
Selecting and implementing overview methods: implications from five exemplar overviews
Authors
Alex Pollock
Pauline Campbell
Ginny Brunton
Harriet Hunt
Lise Estcourt
Publication date
01-12-2017
Publisher
BioMed Central
Published in
Systematic Reviews / Issue 1/2017
Electronic ISSN: 2046-4053
DOI
https://doi.org/10.1186/s13643-017-0534-3

Other articles of this Issue 1/2017

Systematic Reviews 1/2017 Go to the issue

Protocol

Paraquat and Parkinson’s disease: a systematic review protocol according to the OHAT approach for hazard identification

Protocol

Effectiveness of current policing-related mental health interventions in England and Wales and Crisis Intervention Teams as a future potential model: a systematic review

Protocol

Costs related to cardiac arrest management: a systematic review protocol

Protocol

Contacting of authors by systematic reviewers: protocol for a cross-sectional study and a survey

Protocol

Identifying effective components for mobile health behaviour change interventions for smoking cessation and service uptake: protocol of a systematic review and planned meta-analysis

Protocol

Government policy interventions to reduce human antimicrobial use: protocol for a systematic review and meta-analysis