Skip to main content
Top
Published in: BMC Medical Research Methodology 1/2013

Open Access 01-12-2013 | Research article

Risk of bias of randomized controlled trials published in orthopaedic journals

Authors: Laura E Chess, Joel Gagnier

Published in: BMC Medical Research Methodology | Issue 1/2013

Login to get access

Abstract

Background

The purpose of this study was to assess the quality of methodology in orthopaedics-related randomized controlled trials (RCTs) published from January 2006 to December 2010 in the top orthopaedic journals based on impact scores from the Thompson ISI citation reports (2010).

Methods

Journals included American Journal of Sports Medicine; Journal of Orthopaedic Research; Journal of Bone and Joint Surgery, American; Spine Journal; and Osteoarthritis and Cartilage. Each RCT was assessed on ten criteria (randomization method, allocation sequence concealment, participant blinding, outcome assessor blinding, outcome measurement, interventionist training, withdrawals, intent to treat analyses, clustering, and baseline characteristics) as having empirical evidence for biasing treatment effect estimates when not performed properly.

Results

A total of 232 RCTs met our inclusion criteria. The proportion of RCTs in published journals fell from 6% in 2006 to 4% in 2010. Forty-nine percent of the criteria were fulfilled across these journals, with 42% of the criteria not being amendable to assessment due to inadequate reporting. The results of our regression revealed that a more recent publication year was significantly associated with more fulfilled criteria (β = 0.171; CI = −0.00 to 0.342; p = 0.051).

Conclusion

In summary, very few studies met all ten criteria. Thus, many of these studies likely have biased estimates of treatment effects. In addition, these journals had poor reporting of important methodological aspects.
Appendix
Available only for authorised users
Literature
1.
go back to reference Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T: The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001, 134: 663-694. 10.7326/0003-4819-134-8-200104170-00012.CrossRefPubMed Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T: The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001, 134: 663-694. 10.7326/0003-4819-134-8-200104170-00012.CrossRefPubMed
2.
go back to reference Mulrow CD, Cook DJ, Davidoff F: Systematic reviews: critical links in the great chain of evidence. Systematic reviews: synthesis of best evidence for health-care decisions. Edited by: Mulrow C, Cook D. 1998, Philadelphia, PA: American College of Physicians Mulrow CD, Cook DJ, Davidoff F: Systematic reviews: critical links in the great chain of evidence. Systematic reviews: synthesis of best evidence for health-care decisions. Edited by: Mulrow C, Cook D. 1998, Philadelphia, PA: American College of Physicians
3.
go back to reference Sackett DL, Richardsom WS, Rosenberg W, Haynes B: Evidence-based medicine: how to practice and teach EBM. 1998, New York, NY: Churchill Livingstone Sackett DL, Richardsom WS, Rosenberg W, Haynes B: Evidence-based medicine: how to practice and teach EBM. 1998, New York, NY: Churchill Livingstone
4.
go back to reference Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273: 408-412. 10.1001/jama.1995.03520290060030.CrossRefPubMed Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273: 408-412. 10.1001/jama.1995.03520290060030.CrossRefPubMed
5.
go back to reference Moher D, Jadad AR, Tugwell P: Assessing the quality of randomized controlled trials: current issues and future directions. Int J Technol Assess Health Care. 1996, 12: 195-208. 10.1017/S0266462300009570.CrossRefPubMed Moher D, Jadad AR, Tugwell P: Assessing the quality of randomized controlled trials: current issues and future directions. Int J Technol Assess Health Care. 1996, 12: 195-208. 10.1017/S0266462300009570.CrossRefPubMed
6.
go back to reference Jadad AR, Moore A, Carroll D, Jenkinson C, Reynold DJ, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Control Clin Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.CrossRefPubMed Jadad AR, Moore A, Carroll D, Jenkinson C, Reynold DJ, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Control Clin Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.CrossRefPubMed
7.
go back to reference Begg CB, Cho MK, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF: Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA. 1996, 76: 637-639.CrossRef Begg CB, Cho MK, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF: Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA. 1996, 76: 637-639.CrossRef
8.
go back to reference Moher D, Schulz KF, Altman D, for the CONSORT Group (Consolidated Standards of Reporting Trials): The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001, 285: 1987-1991. 10.1001/jama.285.15.1987.CrossRefPubMed Moher D, Schulz KF, Altman D, for the CONSORT Group (Consolidated Standards of Reporting Trials): The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001, 285: 1987-1991. 10.1001/jama.285.15.1987.CrossRefPubMed
10.
go back to reference Ioannidis JPA, Evans SJW, Gøtzsche PC, O’Neill RT, Altman DG, Schulz K, Moher D, for the CONSORT Group: Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Int Med. 2004, 141: 781-788. 10.7326/0003-4819-141-10-200411160-00009.CrossRefPubMed Ioannidis JPA, Evans SJW, Gøtzsche PC, O’Neill RT, Altman DG, Schulz K, Moher D, for the CONSORT Group: Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Int Med. 2004, 141: 781-788. 10.7326/0003-4819-141-10-200411160-00009.CrossRefPubMed
12.
go back to reference Dulai SK, Slobogean BL, Beauchamp RD, Mulpuri K: A quality assessment of randomized clinical trials in pediatric orthopaedics. J Pediatr Orthop. 2007, 27 (5): 573-581. 10.1097/bpo.0b013e3180621f3e.CrossRefPubMed Dulai SK, Slobogean BL, Beauchamp RD, Mulpuri K: A quality assessment of randomized clinical trials in pediatric orthopaedics. J Pediatr Orthop. 2007, 27 (5): 573-581. 10.1097/bpo.0b013e3180621f3e.CrossRefPubMed
13.
go back to reference Bhandari M, Richards RR, Sprague S, Schemitsch EH: The quality of reporting of randomized trials in the journal of bone and joint surgery from 1998 through 2000. J Bone Joint Surg Am. 2002, 84A (3): 388-396. Bhandari M, Richards RR, Sprague S, Schemitsch EH: The quality of reporting of randomized trials in the journal of bone and joint surgery from 1998 through 2000. J Bone Joint Surg Am. 2002, 84A (3): 388-396.
14.
go back to reference Herman A, Boster IB, Tenenbaum S, Chechick A: Intention-to-treat analysis and accounting for missing data in orthopaedic randomized clinical trials. J Bone Joint Surg Am. 2009, 91: 2137-2143. 10.2106/JBJS.H.01481.CrossRefPubMed Herman A, Boster IB, Tenenbaum S, Chechick A: Intention-to-treat analysis and accounting for missing data in orthopaedic randomized clinical trials. J Bone Joint Surg Am. 2009, 91: 2137-2143. 10.2106/JBJS.H.01481.CrossRefPubMed
15.
go back to reference Karanicolas PJ, Bhandari M, Teromi B, Akl EA, Bassler D, Alonso-Colello P, Rigau D, Bryant D, Smith SE, Walter SD, Guyatt GH: Blinding of outcomes in trials of orthopaedic trauma: an opportunity to enhance the validity of clinical trials. J Bone Joint Surg Am. 2008, 90: 1026-1033. 10.2106/JBJS.G.00963.CrossRefPubMed Karanicolas PJ, Bhandari M, Teromi B, Akl EA, Bassler D, Alonso-Colello P, Rigau D, Bryant D, Smith SE, Walter SD, Guyatt GH: Blinding of outcomes in trials of orthopaedic trauma: an opportunity to enhance the validity of clinical trials. J Bone Joint Surg Am. 2008, 90: 1026-1033. 10.2106/JBJS.G.00963.CrossRefPubMed
16.
go back to reference Chan S, Bhandari M: The quality of reporting of orthopaedic randomized trials with use of a checklist for non-pharmacological therapies. J Bone Joint Surg Am. 2007, 89: 1970-1978. 10.2106/JBJS.F.01591.CrossRefPubMed Chan S, Bhandari M: The quality of reporting of orthopaedic randomized trials with use of a checklist for non-pharmacological therapies. J Bone Joint Surg Am. 2007, 89: 1970-1978. 10.2106/JBJS.F.01591.CrossRefPubMed
17.
go back to reference Moher D, Jones A, Lepage L, for the CONSORT Group: Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA. 2001, 285 (15): 1992-1995. 10.1001/jama.285.15.1992.CrossRefPubMed Moher D, Jones A, Lepage L, for the CONSORT Group: Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA. 2001, 285 (15): 1992-1995. 10.1001/jama.285.15.1992.CrossRefPubMed
18.
go back to reference Montane E, Vallano A, Vidal X, Aguilera C, Laporte J: Reporting randomized clinical trials of analgesics after traumatic or orthopaedic surgery is inadequate: a systematic review. BMC Clin Pharmacol. 2010, 10: 2-CrossRefPubMedPubMedCentral Montane E, Vallano A, Vidal X, Aguilera C, Laporte J: Reporting randomized clinical trials of analgesics after traumatic or orthopaedic surgery is inadequate: a systematic review. BMC Clin Pharmacol. 2010, 10: 2-CrossRefPubMedPubMedCentral
19.
go back to reference Parsons NR, Hiskens R, Price CL, Achten J, Costa ML: A systematic review of the quality of research reporting in general orthopaedic journals. J Bone Joint Surg Br. 2011, 93B: 1154-1159.CrossRef Parsons NR, Hiskens R, Price CL, Achten J, Costa ML: A systematic review of the quality of research reporting in general orthopaedic journals. J Bone Joint Surg Br. 2011, 93B: 1154-1159.CrossRef
20.
go back to reference Schulz KF, Altman DG, Moher D: CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med. 2010, 152: 726-732. 10.7326/0003-4819-152-11-201006010-00232.CrossRefPubMed Schulz KF, Altman DG, Moher D: CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med. 2010, 152: 726-732. 10.7326/0003-4819-152-11-201006010-00232.CrossRefPubMed
21.
go back to reference von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP, for the STROBE Initiative: The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007, 370: 1453-1457. 10.1016/S0140-6736(07)61602-X.CrossRefPubMed von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP, for the STROBE Initiative: The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007, 370: 1453-1457. 10.1016/S0140-6736(07)61602-X.CrossRefPubMed
22.
go back to reference Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW: Standards of reporting of randomized controlled trials in general surgery: can we do better?. Ann Surg. 2006, 244 (5): 663-667. 10.1097/01.sla.0000217640.11224.05.CrossRefPubMedPubMedCentral Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW: Standards of reporting of randomized controlled trials in general surgery: can we do better?. Ann Surg. 2006, 244 (5): 663-667. 10.1097/01.sla.0000217640.11224.05.CrossRefPubMedPubMedCentral
23.
go back to reference Juni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282 (11): 1054-1060. 10.1001/jama.282.11.1054.CrossRefPubMed Juni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282 (11): 1054-1060. 10.1001/jama.282.11.1054.CrossRefPubMed
24.
go back to reference Juni P, Tallon D, Egger M: Proceedings of the 3rd symposium on systematic reviews: beyond the basics. St. Catherine’s College. “Garbage in – garbage out?” Assessment of the quality of controlled trials in meta-analyses published in leading journals. 2000, Oxford: Centre for Statistics in Medicine, 19- Juni P, Tallon D, Egger M: Proceedings of the 3rd symposium on systematic reviews: beyond the basics. St. Catherine’s College. “Garbage in – garbage out?” Assessment of the quality of controlled trials in meta-analyses published in leading journals. 2000, Oxford: Centre for Statistics in Medicine, 19-
25.
go back to reference Kjaergard LI, Villumsen J, Gluud C: Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Int Med. 2001, 135 (11): 982-989. 10.7326/0003-4819-135-11-200112040-00010.CrossRefPubMed Kjaergard LI, Villumsen J, Gluud C: Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Int Med. 2001, 135 (11): 982-989. 10.7326/0003-4819-135-11-200112040-00010.CrossRefPubMed
26.
go back to reference Verhagen AP, de Bie R, Lenssen AF, de Vet HC, Kessles AG, Boers M, van den Brandt PA: Impact of quality items on study outcome. Int J Technol Assess Health Care. 2000, 16 (4): 1136-1146. 10.1017/S0266462300103174.CrossRefPubMed Verhagen AP, de Bie R, Lenssen AF, de Vet HC, Kessles AG, Boers M, van den Brandt PA: Impact of quality items on study outcome. Int J Technol Assess Health Care. 2000, 16 (4): 1136-1146. 10.1017/S0266462300103174.CrossRefPubMed
27.
go back to reference Ioannidis JP, Polycarpou A, Ntais C, Pavlidis N: Randomised trials comparing chemotherapy regimens for advanced non-small cell lung cancer: biases and evolution over time. Eur J Cancer. 2003, 39: 2278-2287. 10.1016/S0959-8049(03)00571-9.CrossRefPubMed Ioannidis JP, Polycarpou A, Ntais C, Pavlidis N: Randomised trials comparing chemotherapy regimens for advanced non-small cell lung cancer: biases and evolution over time. Eur J Cancer. 2003, 39: 2278-2287. 10.1016/S0959-8049(03)00571-9.CrossRefPubMed
28.
go back to reference Chalmers TC, Celano P, Sacks HS, Smith H: Bias in treatment assignment in controlled clinical trials. N Engl J Med. 1983, 309: 1358-1361. 10.1056/NEJM198312013092204.CrossRefPubMed Chalmers TC, Celano P, Sacks HS, Smith H: Bias in treatment assignment in controlled clinical trials. N Engl J Med. 1983, 309: 1358-1361. 10.1056/NEJM198312013092204.CrossRefPubMed
29.
go back to reference Kunz R, Oxman AD: The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ. 1998, 317 (7167): 1185-1190. 10.1136/bmj.317.7167.1185.CrossRefPubMedPubMedCentral Kunz R, Oxman AD: The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ. 1998, 317 (7167): 1185-1190. 10.1136/bmj.317.7167.1185.CrossRefPubMedPubMedCentral
30.
go back to reference Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, Contopoulos-Ioannidis DG, Lau J: Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA. 2001, 286 (7): 821-830. 10.1001/jama.286.7.821.CrossRefPubMed Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, Contopoulos-Ioannidis DG, Lau J: Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA. 2001, 286 (7): 821-830. 10.1001/jama.286.7.821.CrossRefPubMed
31.
go back to reference Karassa FB, Tatsioni A, Ioannidis JP: Design, quality, and bias in randomized controlled trials of systemic lupus erythematosus. J Rheumatol. 2003, 30 (5): 979-984.PubMed Karassa FB, Tatsioni A, Ioannidis JP: Design, quality, and bias in randomized controlled trials of systemic lupus erythematosus. J Rheumatol. 2003, 30 (5): 979-984.PubMed
33.
go back to reference Turk DC, Rudy TE: Neglected factors in chronic pain treatment outcome studies—referral patterns, failure to enter treatment, and attrition. Pain. 1990, 43: 7-25. 10.1016/0304-3959(90)90046-G.CrossRefPubMed Turk DC, Rudy TE: Neglected factors in chronic pain treatment outcome studies—referral patterns, failure to enter treatment, and attrition. Pain. 1990, 43: 7-25. 10.1016/0304-3959(90)90046-G.CrossRefPubMed
34.
go back to reference Turk DC, Rudy TE, Sorkin BA: Neglected factors in chronic pain treatment outcome studies: determination of success. Pain. 1993, 53: 3-16. 10.1016/0304-3959(93)90049-U.CrossRefPubMed Turk DC, Rudy TE, Sorkin BA: Neglected factors in chronic pain treatment outcome studies: determination of success. Pain. 1993, 53: 3-16. 10.1016/0304-3959(93)90049-U.CrossRefPubMed
35.
go back to reference Huwiler-Muntener K, Juni P, Junker C, Egger M: Quality of reporting as a measure of methodology quality. JAMA. 2002, 287 (21): 2801-2804. 10.1001/jama.287.21.2801.CrossRefPubMed Huwiler-Muntener K, Juni P, Junker C, Egger M: Quality of reporting as a measure of methodology quality. JAMA. 2002, 287 (21): 2801-2804. 10.1001/jama.287.21.2801.CrossRefPubMed
36.
go back to reference Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG: CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomized trials. BMJ. 2010, 340: c869-10.1136/bmj.c869.CrossRefPubMedPubMedCentral Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG: CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomized trials. BMJ. 2010, 340: c869-10.1136/bmj.c869.CrossRefPubMedPubMedCentral
37.
go back to reference Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gaboury I: Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006, 185 (5): 263-267.PubMed Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gaboury I: Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006, 185 (5): 263-267.PubMed
38.
go back to reference McCulloch P, Altman DG, Campbell WB, Flum DR, Glasziou P, Marshal JC, Nichol J, for the Balliol Collaboration: No surgical innovation without evaluation: the IDEAL recommendations. Lancet. 2009, 374: 1105-1112. 10.1016/S0140-6736(09)61116-8.CrossRefPubMed McCulloch P, Altman DG, Campbell WB, Flum DR, Glasziou P, Marshal JC, Nichol J, for the Balliol Collaboration: No surgical innovation without evaluation: the IDEAL recommendations. Lancet. 2009, 374: 1105-1112. 10.1016/S0140-6736(09)61116-8.CrossRefPubMed
39.
go back to reference Poolman RW, Struijs PA, Krips R, Sierevelt IN, Lutz KH, Bhandari M: Does a “level I evidence” rating imply high quality of reporting in orthopaedic randomised controlled trials?. BMC Med Res Methodol. 2006, 6: 44-10.1186/1471-2288-6-44.CrossRefPubMedPubMedCentral Poolman RW, Struijs PA, Krips R, Sierevelt IN, Lutz KH, Bhandari M: Does a “level I evidence” rating imply high quality of reporting in orthopaedic randomised controlled trials?. BMC Med Res Methodol. 2006, 6: 44-10.1186/1471-2288-6-44.CrossRefPubMedPubMedCentral
Metadata
Title
Risk of bias of randomized controlled trials published in orthopaedic journals
Authors
Laura E Chess
Joel Gagnier
Publication date
01-12-2013
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2013
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-13-76

Other articles of this Issue 1/2013

BMC Medical Research Methodology 1/2013 Go to the issue