Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2016

Open Access 01-12-2016 | Research article

The art and science of study identification: a comparative analysis of two systematic reviews

Authors: Laura Rosen, Ruth Suhami

Published in: BMC Medical Research Methodology | Issue 1/2016

Login to get access

Abstract

Background

Systematic reviews (SRs) form the foundation for guidelines and evidence-based policy in medicine and public health. Although similar systematic reviews may include non-identical sets of studies, and it is recognized that different sets of studies may lead to different conclusions, little work has been published on why SR study cohorts differ.

Methods

We took advantage of concurrent publication of two SRs on the same topic – prevention of child exposure to tobacco smoke - to understand why study cohorts differed in the two reviews. We identified all studies included in just one review, investigated validity of specified reasons for exclusions, and, using database records, explored reasons for study non-identification. We assessed review methods and discordancy, and attempted to assess whether changes in study cohorts would have changed conclusions.

Results

Sixty-one studies were included in the two reviews. Thirty-five studies were present in just one review; of these, twenty were identified and excluded by the parallel review.
Omissions were due to: review scope (9 studies, 26 %), outcomes of interest not measured (8 studies, 23 %), exclusion of reports with inadequate reporting (6 studies, 17 %), mixed or unclear reasons (3 studies, 8 %), search strategies concerning filters, tagging, and keywords (3 studies, 8 %), search strategies regarding sources (PUBMED not searched) (2 studies, 6 %); discordant interpretation of same eligibility criteria (2 studies, 6 %), and non-identification due to non-specific study topic (2 studies, 6 %). Review conclusions differed, but were likely due to differences in synthesis methods, not differences in study cohorts.

Conclusions

The process of study identification for SRs is part art and part science. While some differences are due to differences in review scope, outcomes measured, or reporting practices, others are caused by search methods or discrepancies in reviewer interpretations. Different study cohorts may or may not be a cause of differing SR results. Completeness of SR study cohorts could be enhanced by 1 – independent identification of studies by at least two reviewers, as recommended by recent guidelines, 2 - searching PUBMED with free-text keywords in addition to MEDLINE to identify recent studies, and 3 - Using validated search filters.
Appendix
Available only for authorised users
Literature
1.
Stegenga J. Is meta-analysis the platinum standard of evidence? Stud Hist Philos Biol Biomed Sci. 2011;42(4):497–507.CrossRefPubMed
2.
3.
Mullen PD, Ramirez G. The promise and pitfalls of systematic reviews. Annu Rev Public Health. 2006;27:81–102.CrossRefPubMed
4.
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354(9193):1896–900.CrossRefPubMed
5.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009;62(10):1006–12.CrossRefPubMed
6.
Hutton B, Salanti G, Chaimani A, Caldwell D, Schmid C, Thorlund K, et al. The Quality of Reporting Methods and Results in Network Meta-Analyses: An Overview of Reviews and Suggestions for Improvement. PLoS One. 2014;9(3):e92508.PubMedCentralCrossRefPubMed
7.
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.PubMedCentralCrossRefPubMed
8.
Whiting P, Savovic J, Higgins J, Caldwell D, Reeves BC, Shea BJ, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.PubMedCentralCrossRefPubMed
9.
Delaney A, Bagshaw SM, Ferland A, Laupland K, Manns B, Doig C. The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal. Crit Care Med. 2007;35(2):589–94.CrossRefPubMed
10.
Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998;280(3):278–80.CrossRefPubMed
11.
12.
Wen J, Ren Y, Wang L, Li Y, Liu Y, Zhou M, et al. The reporting quality of meta-analyses improves: a random sampling study. J Clin Epidemiol. 2008;61(8):770–5.CrossRefPubMed
13.
Willis BH, Quigley M. The assessment of the quality of reporting of meta-analyses in diagnostic research: a systematic review. BMC Med Res Methodol. 2011;11:163.PubMedCentralCrossRefPubMed
14.
15.
Jadad AR, Cook DJ, Browman J. A guide to interpreting discordant systematic reviews. Can Med Assoc J. 1997;156:1411–6.
16.
Moja L, del Rio MP F, Banzi R, Cusi C, D'Amico R, Liberati A, et al. Multiple systematic reviews: methods for assessing discordances of results. Intern Emerg Med. 2012;7(6):563–8.CrossRefPubMed
17.
Alperson SY, Berger VW. Opposing systematic reviews: the effects of two quality rating instruments on evidence regarding t'ai chi and bone mineral density in postmenopausal women. J Altern Complement Med. 2011;17(5):389–95.PubMedCentralCrossRefPubMed
18.
Campbell J, Bellamy N, Gee T. Differences between systematic reviews/meta-analyses of hyaluronic acid/hyaluronan/hylan in osteoarthritis of the knee. Osteoarthritis Cartilage. 2007;15(12):1424–36.CrossRefPubMed
19.
Cook DJ, Reeve BK, Guyatt GH, Heyland DK, Griffith LE, Buckingham L, et al. Stress ulcer prophylaxis in critically ill patients. Resolving discordant meta-analyses. JAMA. 1996;275(4):308–14.CrossRefPubMed
20.
Katerndahl DA, Lawler WR. Variability in meta-analytic results concerning the value of cholesterol reduction in coronary heart disease: a meta-meta-analysis. Am J Epidemiol. 1999;149(5):429–41.CrossRefPubMed
21.
22.
Rosen LJ, Ben Noach M. Systematic reviews on tobacco control from Cochrane and the Community Guide: different methods, similar findings. J Clin Epidemiol. 2010;63(6):596–606.CrossRefPubMed
23.
Thacker SB, Stroup DF. Methods and interpretation in systematic reviews: commentary on two parallel reviews of epidural analgesia during labor. Am J Obstet Gynecol. 2002;186(5 Suppl Nature):S78–80.CrossRefPubMed
24.
Bown MJ, Sutton AJ. Quality control in systematic reviews and meta-analyses. Eur J Vasc Endovasc Surg. 2010;40(5):669–77.CrossRefPubMed
25.
Ford A, Guyatt G, Talley N, Moayyedi P. Errors in the conduct of systematic reviews of pharmacological interventions for irritable bowel syndrome. Am J Gastroenterol. 2010;105(2):280–8.CrossRefPubMed
26.
Higgins JPT, Green S E. Cochrane Handbook for Systematic Reviews of Interventions 4.2.6 [updated September 2006]. Chichester, UK: The Cochrane Library, Issue 4. John Wiley & Sons, Ltd.; 2006.
27.
Higgins J, Green S (editors): Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. In.: Cochrane Collaboration; 2011.
28.
Goodyear‐Smith F, van Driel M, Arroll B, Del Mar C. Analysis of decisions made in meta-analyses of depression screening and the risk of confirmation bias: A case study. BMC Med Res Methodol. 2012;12:76.PubMedCentralCrossRefPubMed
29.
Rosen LJ, Myers V, Hovell M, Zucker D, Ben Noach M. Meta-analysis of parental protection of children from tobacco smoke exposure. Pediatrics. 2014;133(4):698–714.CrossRefPubMed
30.
Baxi R, Sharma M, Roseby R, Polnay A, Priest N, Waters E, et al. Family and carer smoking control programmes for reducing children's exposure to environmental tobacco smoke. Cochrane Database Syst Rev. 2014;3, CD001746.PubMed
31.
Andrews J, Guyatt GH, Oxman AD. GRADE guideslines: 14. Going from evidence to recommendations: the significance and presentation of recommendations. J Clin Epidemiol. 2013;66(7):719–25.CrossRefPubMed
32.
Rosen LJ, Noach MB, Winickoff JP, Hovell MF. Parental smoking cessation to protect young children: a systematic review and meta-analysis. Pediatrics. 2012;129(1):141–52.CrossRefPubMed
33.
Rosen LJ. E-letter: Revised tables re: Parental Smoking Cessation to Protect Young Children: A Systematic Review and Meta-analysis. Pediatrics. 2013.
34.
Stotts A, Northrup T, Green C, Evans P, Tyson J, Hovell M: The Baby's Breath Project: A pilot trial to reduce secondhand smoke exposure in high respiratory risk infants in the neonatal intensive care unit (POS1-69). In: Society for Research on Nicotine and Tobacco 18th Annual Meeting March 13-16, 2012, Houston, Texas. 2012: 60.
35.
Stotts AL, Green C, Northrup TF, Dodrill CL, Evans P, Tyson J, et al. Feasibility and efficacy of an intervention to reduce secondhand smoke exposure among infants discharged from a neonatal intensive care unit. J Perinatol. 2013;33(10):811–6.CrossRefPubMed
36.
Armstrong KL, Fraser JA, Dadds MR, Morris J. Promoting secure attachment, maternal mood and child health in a vulnerable population: A randomized controlled trial. J Paediatr Child Health. 2000;36(6):555–62.CrossRefPubMed
37.
Borrelli B, McQuaid EL, Novak SP, Hammond SK, Becker B. Motivating Latino Caregivers of Children With Asthma to Quit Smoking: A Randomized Trial. J Consult Clin Psychol. 2010;78(1):34–43.CrossRefPubMed
38.
Chan SSC, Lam TH, Salili F, Leung GM, Wong DCN, Botelho RJ, et al. A randomized controlled trial of an individualized motivational intervention on smoking cessation for parents of sick children: A pilot study. Appl Nurs Res. 2005;18(3):178–81.CrossRefPubMed
39.
Chilmonczyk BA, Palomaki GE, Knight GJ, Williams J, Haddow JE. An unsuccessful cotinine-assisted intervention strategy to reduce environmental tobacco smoke exposure during infancy. Am J Dis Child. 1992;146(3):357–60.PubMed
40.
Culp AM, Culp RE, Anderson JW, Carter S. Health and safety intervention with first-time mothers. Health Educ Res. 2007;22(2):285–94.CrossRefPubMed
41.
Curry SJ, Ludman EJ, Graham E, Stout J, Grothaus L, Lozano P. Pediatric-based smoking cessation intervention for low-income women: A randomized trial. Arch Pediatr Adolesc Med. 2003;157(3):295–302.CrossRefPubMed
42.
Davis SW, Cummings KM, Rimer BK, Sciandra R, Stone JC. The impact of tailored self-help smoking cessation guides on young mothers. Health Educ Q. 1992;19(4):495–504.CrossRefPubMed
43.
Ekerbicer HC, Celik M, Guler E, Davutoglu M, Kilinc M. Evaluating environmental tobacco smoke exposure in a Group of turkish primary school students and developing intervention methods for prevention. BMC Public Health. 2007;7.
44.
Elder JP, Perry CL, Stone EJ, Johnson CC, Yang M, Edmundson EW, et al. Tobacco use measurement, prediction, and intervention in elementary schools in four states: The CATCH study. Prev Med. 1996;25(4):486–94.CrossRefPubMed
45.
Emmons KM, Hammond SK, Fava JL, Velicer WF, Evans JL, Monroe AD. A randomized trial to reduce passive smoke exposure in low-income households with young children. Pediatrics. 2001;108(1):18–24.CrossRefPubMed
46.
French GM, Groner JA, Wewers ME, Ahijevych K. Staying smoke free: An intervention to prevent postpartum relapse. Nicotine Tob Res. 2007;9(6):663–70.CrossRefPubMed
47.
Hannover W, Thyrian JR, Roske K, Grempler J, Rumpf HJ, John U, et al. Smoking cessation and relapse prevention for postpartum women: Results from a randomized controlled trial at 6, 12, 18 and 24 months. Addict Behav. 2009;34(1):1–8.CrossRefPubMed
48.
Herbert RJ, Gagnon AJ, O'Loughlin JL, Rennick JE. Testing an empowerment intervention to help parents make homes smoke-free: a randomized controlled trial. J Community Health. 2011;36(4):650–7.CrossRefPubMed
49.
Huang CM, Wu HL, Huang SH, Chien LY, Guo JL. Transtheoretical model-based passive smoking prevention programme among pregnant women and mothers of young children. Eur J Public Health. 2013;23(5):777–82.CrossRefPubMed
50.
Hughes DM, McLeod M, Garner B, Goldbloom RB. Controlled trial of a home and ambulatory program for asthmatic children. Pediatrics. 1991;87(1):54–61.PubMed
51.
Kallio K, Jokinen E, Hamalainen M, Kaitosaari T, Volanen I, Viikari J, et al. Impact of repeated lifestyle counselling in an atherosclerosis prevention trial on parental smoking and children's exposure to tobacco smoke. Acta Paediatr, Int J Paediatr. 2006;95(3):283–90.CrossRef
52.
Kimata H. Cessation of passive smoking reduces allergic responses and plasma neurotrophin. Eur J Clin Invest. 2004;34(2):165–6.CrossRefPubMed
53.
Lanphear BP, Hornung RW, Khoury J, Yolton K, Lierl M, Kalkbrenner A. Effects of HEPA air cleaners on unscheduled asthma visits and asthma symptoms for children exposed to secondhand tobacco smoke. Pediatrics. 2011;127(1):93–101.PubMedCentralCrossRefPubMed
54.
Nuesslein TG, Struwe A, Maiwald N, Rieger C, Stephan V. Maternal tobacco consumption can be reduced by simple intervention of the paediatrician. Klin Padiatr. 2006;218(5):283–6.CrossRefPubMed
55.
Patel S, Hendry P, Kalynych C, Butterfield R, Lott M, Lukens-Bull K. The impact of third-hand smoke education in a pediatric emergency department on caregiver smoking policies and quit status: A pilot study. Int J Disabil Human Dev. 2012;11(4):335–42.CrossRef
56.
Phillips RM, Merritt TA, Goldstein MR, Deming DD, Slater LE, Angeles DM. Prevention of postpartum smoking relapse in mothers of infants in the neonatal intensive care unit. J Perinatol. 2012;32(5):374–80.PubMedCentralCrossRefPubMed
57.
Pulley KR, Flanders-Stepans MB. Smoking hygiene: an educational intervention to reduce respiratory symptoms in breastfeeding infants exposed to tobacco. J Perinat Educ. 2002;11(3):28–37.PubMedCentralPubMed
58.
Ralston S, Roohi M. A randomized, controlled trial of smoking cessation counseling provided during child hospitalization for respiratory illness. Pediatr Pulmonol. 2008;43(6):561–6.CrossRefPubMed
59.
Ralston S, Grohman C, Word D, Williams J. A randomized trial of a brief intervention to promote smoking cessation for parents during child hospitalization. Pediatr Pulmonol. 2013;48(6):608–13.CrossRefPubMed
60.
Ratner PA, Johnson JL, Bottorff JL. Mothers' efforts to protect their infants from environmental tobacco smoke. Can J Public Health. 2001;92(1):46–7.PubMed
61.
Schonberger HJAM, Dompeling E, Knottnerus JA, Maas T, Muris JWM, van Weel C, et al. The PREVASC study: The clinical effect of a multifaceted educational intervention to prevent childhood asthma. Eur Respir J. 2005;25(4):660–70.CrossRefPubMed
62.
Streja L, Crespi CM, Bastani R, Wong GC, Jones CA, Bernert JT, et al. Can a minimal intervention reduce secondhand smoke exposure among children with asthma from low income minority families? Results of a randomized trial. J Immigr Minor Health. 2014;16(2):256–64.PubMedCentralCrossRefPubMed
63.
Teach SJ, Crain EF, Quint DM, Hylan ML, Joseph JG: Improved asthma outcomes in a high-morbidity pediatric population: results of an emergency department-based randomized clinical trial. Arch Pediatr Adolesc Med, 160(5):535-41.
64.
Van't Hof SM, Wall MA, Dowler DW, Stark MJ. Randomised controlled trial of a postpartum relapse prevention intervention. Tob Control. 2000;9 Suppl 3:III64–6.
65.
Vineis P, Ronco G, Ciccone G, Vernero E, Troia B, D'Incalci T, et al. Prevention of exposure of young children to parental tobacco smoke: Effectiveness of an educational program. Tumori. 1993;79(3):183–6.PubMed
66.
Wahlgren DR, Hovell MF, Meltzer SB, Hofstetter CR, Zakarian JM. Reduction of environmental tobacco smoke exposure in asthmatic children: A 2-year follow-up. Chest. 1997;111(1):81–8.CrossRefPubMed
67.
Wiggins M, Oakley A, Roberts I, Turner H, Rajan L, Austerberry H, et al. Postnatal support for mothers living in disadvantaged inner city areas: A randomised controlled trial. J Epidemiol Community Health. 2005;59(4):288–95.PubMedCentralCrossRefPubMed
68.
Winickoff JP, Healey EA, Regan S, Park ER, Cole C, Friebely J, et al. Using the postpartum hospital stay to address mothers' and fathers' smoking: The NEWS study. Pediatrics. 2010;125(3):518–25.CrossRefPubMed
69.
Woodward A, Owen N, Grgurinovich N, Griffith F, Linke H. Trial of an intervention to reduce passive smoking in infancy. Pediatr Pulmonol. 1987;3(3):173–8.CrossRefPubMed
70.
Zhang D, Qiu X. School-based tobacco-use prevention - People's Republic of China, May 1989-January 1990. JAMA. 1993;269(23):2972.CrossRef
71.
Hovell MF, Meltzer SB, Zakarian JM, Wahlgren DR, Emerson JA, Hofstetter CR, et al. Reduction of environmental tobacco smoke exposure among asthmatic children: a controlled trial. Chest. 1994;106(2):440–6.CrossRefPubMed
72.
Borenstein M, Hedges L, Higgins J, Rothstein H. Introduction to Meta-Analysis. West Sussix, UK: Wiley; 2009.CrossRef
73.
74.
75.
76.
Duffy S, Misso K, Noake C, Ross J, L. S: Supplementary searches of PubMed to improve currency of MEDLINE and MEDLINE In-Process searches via OvidSP. Kleijnen Systematic Reviews Ltd, York. In: UK InterTASC Information Specialists’ Sub-Group (ISSG) Workshop. Exeter: UK; 2014.
77.
Crumley ET, Wiebe N, Cramer K, Klassen TP, Hartling L. Which resources should be used to identify RCT/CCTs for systematic reviews: a systematic review. BMC Med Res Methodol. 2005;5:24.PubMedCentralCrossRefPubMed
78.
79.
McKibbon KA, Wilczynski NL, Haynes RB. Retrieving randomized controlled trials from medline: a comparison of 38 published search filters. Health Inf Libr J. 2009;26(3):187–202.CrossRef
80.
Jenkins M. Evaluation of methodological search filters—a review. Health Inf Libr J. 2004;21(3):148–63.CrossRef
81.
Harbour J, Fraser C, Lefebvre C, Glanville J, Beale S, Boachie C, et al. Reporting methodological search filter performance comparisons: a literature review. Health Inf Libr J. 2014;31(3):176–94.CrossRef
82.
Metadata
Title
The art and science of study identification: a comparative analysis of two systematic reviews
Authors
Laura Rosen
Ruth Suhami
Publication date
01-12-2016
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2016
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-016-0118-2

Other articles of this Issue 1/2016

BMC Medical Research Methodology 1/2016 Go to the issue

Research article

Specialist nurses’ perceptions of inviting patients to participate in clinical research studies: a qualitative descriptive study of barriers and facilitators

Research article

Bias and precision of methods for estimating the difference in restricted mean survival time from an individual patient data meta-analysis

Research article

Potential use of telephone surveys for non-communicable disease surveillance in developing countries: evidence from a national household survey in Lebanon

Research article

Population cardiovascular health and urban environments: the Heart Healthy Hoods exploratory study in Madrid, Spain

Research article

A consumer register: an acceptable and cost-effective alternative for accessing patient populations

Research article

A comparison of time dependent Cox regression, pooled logistic regression and cross sectional pooling with simulations and an application to the Framingham Heart Study