Top

BMC Medical Research Methodology

Published in:

Open Access 01-12-2014 | Research article

Impact of adding a limitations section to abstracts of systematic reviews on readers’ interpretation: a randomized controlled trial

Authors: Amélie Yavchitz, Philippe Ravaud, Sally Hopewell, Gabriel Baron, Isabelle Boutron

Published in: BMC Medical Research Methodology | Issue 1/2014

Abstract

Background

To allow an accurate evaluation of abstracts of systematic reviews, the PRISMA Statement recommends that the limitations of the evidence (e.g., risk of bias, publication bias, inconsistency, imprecision) should be described in the abstract. We aimed to evaluate the impact of adding such limitations sections on reader’s interpretation.

Method

We performed a two-arm parallel group randomized controlled trial (RCT) using a sample of 30 abstracts of systematic reviews evaluating the effects of healthcare intervention with conclusions favoring the beneficial effect of the experimental treatments. Two formats of these abstracts were derived: one reported without and one with a standardized limitations section written according to the PRISMA statement for abstracts. The primary outcome was readers’ confidence in the results of the systematic review as stated in the abstract assessed by a Likert scale from 0, not at all confident, to 10, very confident. In total, 300 participants (corresponding authors of RCT reports indexed in PubMed) were randomized by a web-based randomization procedure to interpret one abstract with a limitations section (n = 150) or without a limitations section (n = 150). Participants were blinded to the study hypothesis.

Results

Adding a limitations section did not modify readers’ interpretation of findings in terms of confidence in the results (mean difference [95% confidence interval] 0.19 [−0.37–0.74], p = 0.50), confidence in the validity of the conclusions (0.07 [−0.49–0.62], p = 0.80), or benefit of the experimental intervention (0.12 [−0.42–0.44], p = 0.65).

This study is limited because the participants were expert-readers and are not representative of all systematic review readers.

Conclusion

Adding a limitations section to abstracts of systematic reviews did not affect readers’ interpretation of the abstract results. Other studies are needed to confirm the results and explore the impact of a limitations section on a less expert panel of participants.

Trial registration

ClinicalTrial.gov (NCT01848782).

Available only for authorised users

Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn’t. BMJ. 1996, 312: 71-72. 10.1136/bmj.312.7023.71.CrossRefPubMedPubMedCentral

Bastian H, Glasziou P, Chalmers I: Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?. PLoS Med. 2010, 7: e1000326-10.1371/journal.pmed.1000326.CrossRefPubMedPubMedCentral

Cook DJ, Mulrow CD, Haynes RB: Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997, 126: 376-380. 10.7326/0003-4819-126-5-199703010-00006.CrossRefPubMed

Bero LA, Jadad AR: How consumers and policymakers can use systematic reviews for decision making. Ann Intern Med. 1997, 127: 37-42. 10.7326/0003-4819-127-1-199707010-00007.CrossRefPubMed

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007, 7: 10-10.1186/1471-2288-7-10.CrossRefPubMedPubMedCentral

Saint S, Christakis DA, Saha S, Elmore JG, Welsh DE, Baker P, Koepsell TD: Journal reading habits of internists. J Gen Intern Med. 2000, 15: 881-884. 10.1046/j.1525-1497.2000.00202.x.CrossRefPubMedPubMedCentral

Scherer RW, Langenberg P, von Elm E: Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007, MR000005-2

The PLoS Medicine Editors: The impact of open access upon public health. PLoS Med. 2006, 3: e252-doi:10.1371/journal.pmed.0030252CrossRefPubMedCentral

Hopewell S, Clarke M, Askie L: Reporting of trials presented in conference abstracts needs to be improved. J Clin Epidemiol. 2006, 59: 681-684. 10.1016/j.jclinepi.2005.09.016.CrossRefPubMed

10.

Beller EM, Glasziou PP, Hopewell S, Altman DG: Reporting of effect direction and size in abstracts of systematic reviews. JAMA. 2011, 306: 1981-1982. 10.1001/jama.2011.1620.CrossRefPubMed

11.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009, 339: b2700-10.1136/bmj.b2700.CrossRefPubMedPubMedCentral

12.

Beller EM, Glasziou PP, Altman DG, Hopewell S, Bastian H, Chalmers I, Gøtzsche PC, Lasserson T, Tovey D: PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts. PLoS Med. 2013, 10: e1001419-10.1371/journal.pmed.1001419.CrossRefPubMedPubMedCentral

13.

Hopewell S, Boutron I, Altman DG, Ravaud P: Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. BMJ Open. 2013, 3: e003342-CrossRefPubMedPubMedCentral

14.

The Editors: Addressing the limitations of structured abstracts. Ann Intern Med. 2004, 140: 480-481.CrossRef

15.

Schulz KF, Altman DG, Moher D: CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol. 2010, 63: 834-840. 10.1016/j.jclinepi.2010.02.005.CrossRefPubMed

16.

Guyatt GH: GrADe: what is “quality of evidence” and why is it important to clinicians?. BMJ. 2008, 336: 995-998. 10.1136/bmj.39490.551019.BE.CrossRefPubMedPubMedCentral

17.

Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Sch\ünemann HJ: GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008, 336: 924-10.1136/bmj.39489.470347.AD.CrossRefPubMedPubMedCentral

18.

Soffer A: Abstracts of clinical investigations. A new and standardized format. Chest. 1987, 92: 389-390. 10.1378/chest.92.3.389.CrossRefPubMed

19.

Huth EJ: Structured abstracts for papers reporting clinical trials. Ann Intern Med. 1987, 106: 626-10.7326/0003-4819-106-4-626.CrossRefPubMed

20.

Hopewell S, Clarke M, Moher D, Wager E, Middleton P, Altman DG, Schulz KF: CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration. PLoS Med. 2008, 5: e20-10.1371/journal.pmed.0050020.CrossRefPubMedPubMedCentral

21.

Hopewell S, Ravaud P, Baron G, Boutron I: Effect of editors’ implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012, 344: e4178-10.1136/bmj.e4178.CrossRefPubMedPubMedCentral

22.

Ioannidis JP, Lau J: Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. JAMA. 2001, 285: 437-443. 10.1001/jama.285.4.437.CrossRefPubMed

23.

Yank V, Rennie D, Bero LA: Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study. BMJ. 2007, 335: 1202-1205. 10.1136/bmj.39376.447211.BE.CrossRefPubMedPubMedCentral

24.

Boutron I, Dutton S, Ravaud P, Altman DG: Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010, 303: 2058-2064. 10.1001/jama.2010.651.CrossRefPubMed

25.

Kesselheim AS, Robertson CT, Myers JA, Rose SL, Gillet V, Ross KM, Glynn RJ, Joffe S, Avorn J: A randomized study of How physicians interpret research funding disclosures. N Engl J Med. 2012, 367: 1119-1127. 10.1056/NEJMsa1202397.CrossRefPubMedPubMedCentral

26.

Bucher HC, Weinbacher M, Gyr K: Influence of method of reporting study results on decision of physicians to prescribe drugs to lower cholesterol concentration. BMJ. 1994, 309: 761-764. 10.1136/bmj.309.6957.761.CrossRefPubMedPubMedCentral

27.

Forrow L, Taylor WC, Arnold RM: Absolutely relative: how research results are summarized can affect treatment decisions. Am J Med. 1992, 92: 121-124. 10.1016/0002-9343(92)90100-P.CrossRefPubMed

The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/14/123/prepub

Title: Impact of adding a limitations section to abstracts of systematic reviews on readers’ interpretation: a randomized controlled trial
Authors: Amélie Yavchitz
Philippe Ravaud
Sally Hopewell
Gabriel Baron
Isabelle Boutron
Publication date: 01-12-2014
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2014
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/1471-2288-14-123

Keynote webinar | Spotlight on sleep in brain health

Springer Medicine

Impact of adding a limitations section to abstracts of systematic reviews on readers’ interpretation: a randomized controlled trial

Abstract

Background

Method

Results

Conclusion

Trial registration

Keynote webinar | Spotlight on sleep in brain health

Springer Medicine

Abstract

Background

Method

Results

Conclusion

Trial registration

Please log in to get access to this content

Other articles of this Issue 1/2014

Three steps to writing adaptive study protocols in the early phase clinical development of new medicines

Comparative efficiency research (COMER): meta-analysis of cost-effectiveness studies

An overview of the statistical methods reported by studies using the Canadian community health survey

Factors influencing recruitment to research: qualitative study of the experiences and perceptions of research teams

Bias in the study of prediction of change: a Monte Carlo simulation study of the effects of selective attrition and inappropriate modeling of regression toward the mean

Assessing discriminative ability of risk models in clustered data