Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2008

Open Access 01-12-2008 | Research article

Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies

Authors: Andreas Lundh, Peter C Gøtzsche

Published in: BMC Medical Research Methodology | Issue 1/2008

Login to get access

Abstract

Background

Assessing the risk of bias in individual studies in a systematic review can be done using individual components or by summarizing the study quality in an overall score.

Methods

We examined the instructions to authors of the 50 Cochrane Review Groups that focus on clinical interventions for recommendations on methodological quality assessment of studies.

Results

Forty-one of the review groups (82%) recommended quality assessment using components and nine using a scale. All groups recommending components recommended to assess concealment of allocation, compared to only two of the groups recommending scales (P < 0.0001). Thirty-five groups (70%) recommended assessment of sequence generation and 21 groups (42%) recommended assessment of intention-to-treat analysis. Only 28 groups (56%) had specific recommendations for using the quality assessment of studies analytically in reviews, with sensitivity analysis, quality as an inclusion threshold and subgroup analysis being the most commonly recommended methods. The scales recommended had problems in the individual items and some of the groups recommending components recommended items not related to bias in their quality assessment.

Conclusion

We found that recommendations by some groups were not based on empirical evidence and many groups had no recommendations on how to use the quality assessment in reviews. We suggest that all Cochrane Review Groups refer to the Cochrane Handbook for Systematic Reviews of Interventions, which is evidence-based, in their instructions to authors and that their own guidelines are kept to a minimum and describe only how methodological topics that are specific to their fields should be handled.
Appendix
Available only for authorised users
Literature
1.
Egger M, Smith GD, Sterne JA: Uses and abuses of meta-analysis. Clin Med. 2001, 1: 478-84.CrossRef
2.
Feinstein AR: An outline from cause-effect evaluations. Clinical Epidemiology: The Architecture of Clinical Research. 1985, Philadelphia: WB Saunders, 39-52.
3.
Pildal J, Hróbjartsson A, Jørgensen KJ, Hilden J, Altman DG, Gøtzsche PC: Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007, 36: 847-57. 10.1093/ije/dym087.CrossRefPubMed
4.
Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273: 408-12. 10.1001/jama.273.5.408.CrossRefPubMed
5.
Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet. 1998, 352: 609-13. 10.1016/S0140-6736(98)01085-X.CrossRefPubMed
6.
Tierney JF, Stewart LA: Investigating patient exclusion bias in meta-analysis. Int J Epidemiol. 2005, 34: 79-87. 10.1093/ije/dyh300.CrossRefPubMed
7.
Kjaergard LL, Villumsen J, Gluud C: Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med. 2001, 135: 982-9.CrossRefPubMed
8.
Egger M, Jüni P, Bartlett C, Holenstein F, Sterne J: How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003, 7: 1-76.PubMed
9.
Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, Pham B, Klassen TP: Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess. 1999, 3: i-iv. 1–98PubMed
10.
Emerson JD, Burdick E, Hoaglin DC, Mosteller F, Chalmers TC: An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Control Clin Trials. 1990, 11: 339-52. 10.1016/0197-2456(90)90175-2.CrossRefPubMed
11.
Jüni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282: 1054-60. 10.1001/jama.282.11.1054.CrossRefPubMed
12.
Higgins JPT, Green S, editors: Assessment of study quality. Cochrane Handbook for Systematic Reviews of Interventions 4.2.6 [updated September 2006]; Section 6. The Cochrane Library. 2006, Chichester, UK: John Wiley & Sons, Ltd, 4
13.
14.
Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, Moher D: Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998, 280: 278-80. 10.1001/jama.280.3.278.CrossRefPubMed
15.
Jørgensen AW, Hilden J, Gøtzsche PC: Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ. 2006, 333: 782-5. 10.1136/bmj.38973.444699.0B.CrossRefPubMedPubMedCentral
16.
Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A: Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ. 2005, 330: 1053-10.1136/bmj.38414.515938.8F.CrossRefPubMedPubMedCentral
17.
18.
Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Control Clin Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.CrossRefPubMed
19.
Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S: Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995, 16: 62-73. 10.1016/0197-2456(94)00031-W.CrossRefPubMed
20.
Bhandari M, Richards RR, Sprague S, Schemitsch EH: Quality in the reporting of randomized trials in surgery: is the Jadad scale reliable?. Control Clin Trials. 2001, 22: 687-8. 10.1016/S0197-2456(01)00147-7.CrossRefPubMed
21.
Clark HD, Wells GA, Huet C, McAlister FA, Salmi LR, Fergusson D, Laupacis A: Assessing the quality of randomized trials: reliability of the Jadad scale. Control Clin Trials. 1999, 20: 448-52. 10.1016/S0197-2456(99)00026-4.CrossRefPubMed
22.
Greenland S: Quality scores are useless and potentially misleading. Am J Epidemiol. 1994, 140: 300-1.
23.
van Tulder M, Furlan A, Bombardier C, Bouter L, Editorial Board of the Cochrane Collaboration Back Review Group: Updated method guidelines for systematic reviews in the cochrane collaboration back review group. Spine. 2003, 28: 1290-9. 10.1097/00007632-200306150-00014.
24.
Jüni P, Altman DG, Egger M: Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001, 323: 42-6. 10.1136/bmj.323.7303.42.CrossRefPubMedPubMedCentral
25.
26.
Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B: Evidence b(i)ased medicine-selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ. 2003, 326: 1171-3. 10.1136/bmj.326.7400.1171.CrossRefPubMedPubMedCentral
27.
Melander H: [Selective reporting-greater problem than selective publishing?]. Läkartidningen. 2005, 102: 224-5.PubMed
28.
Altman DG: Comparability of randomised groups. Statistician. 1985, 34: 125-36. 10.2307/2987510.CrossRef
29.
Schulz KF, Chalmers I, Grimes DA, Altman DG: Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. JAMA. 1994, 272: 125-8. 10.1001/jama.272.2.125.CrossRefPubMed
30.
Moncrieff J, Churchill R, Drummond DC, McGuire H: Development of a quality assessment instrument for trials of treatment for depression and neurosis. Int J Methods Psychiatr Res. 2001, 10: 126-33. 10.1002/mpr.108.CrossRef
31.
32.
Gøtzsche PC, Hróbjartsson A: Somatostatin analogues for acute bleeding oesophageal varices. Cochrane Database Syst Rev. 2005, CD000193-1
33.
34.
Kunz R, Vist G, Oxman AD: Randomisation to protect against selection bias in healthcare trials. Cochrane Database Syst Rev. 2007, MR000012-2
35.
Deeks JJ: Systematic reviews evaluating effects of health care interventions: Issues of synthesis and bias. [Dissertation]. 2007, University of Birmingham
36.
Stampfer MJ, Colditz GA: Estrogen replacement therapy and coronary heart disease: a quantitative assessment of the epidemiologic evidence. Prev Med. 1991, 20: 47-63. 10.1016/0091-7435(91)90006-P.CrossRefPubMed
37.
Stampfer MJ, Colditz GA, Willett WC, Manson JE, Rosner B, Speizer FE, Hennekens CH: Postmenopausal estrogen therapy and cardiovascular disease. Ten-year follow-up from the nurses' health study. N Engl J Med. 1991, 325: 756-62.CrossRefPubMed
38.
Rossouw JE, Anderson GL, Prentice RL, LaCroix AZ, Kooperberg C, Stefanick ML, Jackson RD, Beresford SA, Howard BV, Johnson KC, Kotchen JM, Ockene J: Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results From the Women's Health Initiative randomized controlled trial. JAMA. 2002, 288: 321-33. 10.1001/jama.288.3.321.CrossRefPubMed
39.
40.
Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, Oxman AD, Phillips B, Schünemann HJ, Edejer TT, Varonen H, Vist GE, Williams JW, Zaza S: Grading quality of evidence and strength of recommendations. BMJ. 2004, 328: 1490-4. 10.1136/bmj.328.7454.1490.CrossRefPubMed
41.
42.
Tugwell P, Shea B, Boers M, Brooks P, Simon L, Strand V, Wells G: Introduction. Evidence-based Rheumatology. London: Blackwell BMJ Books. 2004, XIII-XXVII.
43.
44.
45.
Gøtzsche PC: Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis. Control Clin Trials. 1989, 10: 31-56. 10.1016/0197-2456(89)90017-2. (amendment: 356).CrossRefPubMed
46.
Chan A-W, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004, 291: 2457-65. 10.1001/jama.291.20.2457.CrossRefPubMed
Metadata
Title
Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies
Authors
Andreas Lundh
Peter C Gøtzsche
Publication date
01-12-2008
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2008
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-8-22

Other articles of this Issue 1/2008

BMC Medical Research Methodology 1/2008 Go to the issue

Research article

Alternative regression models to assess increase in childhood BMI

Research article

Intervention, recruitment and evaluation challenges in the Bangladeshi community: Experience from a peer lead educational course

Technical advance

Estimates of adherence and error analysis of physical activity data collected via accelerometry in a large study of free-living adults

Research article

Sampling 'hard-to-reach' populations in health research: yield from a study targeting Americans living in Canada

Research article

The harvest plot: A method for synthesising evidence about the differential effects of interventions

Research article

Bayes rules for optimally using Bayesian hierarchical regression models in provider profiling to identify high-mortality hospitals