Top

Clinical Orthopaedics and Related Research®

Published in:

01-11-2013 | Clinical Research

Have Levels of Evidence Improved the Quality of Orthopaedic Research?

Authors: Brian P. Cunningham, MD, Samuel Harmsen, MD, Chris Kweon, MD, Jason Patterson, MD, Robert Waldrop, MD, Alex McLaren, MD, Ryan McLemore, PhD

Published in: Clinical Orthopaedics and Related Research® | Issue 11/2013

Abstract

Background

Since 2003 many orthopaedic journals have adopted grading systems for levels of evidence (LOE). It is unclear if the quality of orthopaedic literature has changed since LOE was introduced.

Questions/purposes

We asked three questions: (1) Have the overall number and proportion of Level I and II studies increased in the orthopaedic literature since the introduction of LOE? (2) Is a similar pattern seen in individual orthopaedic subspecialty journals? (3) What is the interobserver reliability of grading LOE?

Methods

We assigned LOE to therapeutic studies published in 2000, 2005, and 2010 in eight major orthopaedic subspecialty journals. Number and proportion of Level I and II publications were determined. Data were evaluated using log-linear models. Twenty-six reviewers (13 residents and 13 attendings) graded LOE of 20 blinded therapeutic articles from the Journal of Bone and Joint Surgery for 2009. Interobserver agreement relative to the Journal of Bone and Joint Surgery was assessed using a weighted kappa.

Results

The total number of Level I and II publications in subspecialty journals increased from 150 in 2000 to 239 in 2010. The proportion of high-quality publications increased with time (p < 0.001). All subspecialty journals other than the Journal of Pediatric Orthopaedics and the Journal of Orthopaedic Trauma showed a similar behavior. Average weighted kappa was 0.791 for residents and 0.842 for faculty (p = 0.209).

Conclusions

The number and proportion of Level I and II publications have increased. LOE can be graded reliably with high interobserver agreement. The number and proportion of high-level studies should continue to increase.

Altman DG. Practical Statistics for Medical Research. 2nd ed. London, England: Chapman & Hall/CRC; 2006.

Bhandari M, Swiontkowski MF, Einhorn TA, Tornetta P 3rd, Schemitsch EH, Leece P, Sprague S, Wright JG. Interobserver agreement in the application of levels of evidence to scientific papers in the American volume of the Journal of Bone and Joint Surgery. J Bone Joint Surg Am. 2004;86:1717–1720.PubMed

Brophy RH, Gardner MJ, Saleem O, Marx RG. An assessment of the methodological quality of research published in The American Journal of Sports Medicine. Am J Sports Med. 2005;33:1812–1815.PubMedCrossRef

Cashin MS, Kelley SP, Douziech JR, Varghese RA, Hamilton QP, Mulpuri K. The levels of evidence in pediatric orthopaedic journals: where are we now? J Pediatr Orthop. 2011;31:721–725.PubMedCrossRef

Chang EY, Pannucci CJ, Wilkins EG. Quality of clinical studies in aesthetic surgery journals: a 10-year review. Aesthet Surg J. 2009;29:144–147; discussion 147–149.

Cochrane AL. Archie Cochrane in his own words. Selections arranged from his 1972 introduction to “Effectiveness and Efficiency: Random Reflections on the Health Services” 1972. Control Clin Trials. 1989;10:428–433.

Donegan DJ, Kim TW, Lee GC. Publication rates of presentations at an annual meeting of the American Academy of Orthopaedic Surgeons. Clin Orthop Relat Res. 2010;468:1428–1435.PubMedCrossRef

Ellis J, Mulligan I, Rowe J, Sackett DL. Inpatient general medicine is evidence based. A-Team, Nuffield Department of Clinical Medicine. Lancet. 1995;346:407–410.PubMedCrossRef

Esses SI, McGuire R, Jenkins J, Finkelstein J, Woodard E, Watters WC 3rd, Goldberg MJ, Keith M, Turkelson CM, Wies JL, Sluka P, Boyer KM, Hitchcock K. The treatment of symptomatic osteoporotic spinal compression fractures. J Am Acad Orthop Surg. 2011;19:176–182.PubMed

10.

Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine. JAMA. 1992;268:2420–2425.CrossRef

11.

Hanzlik S, Mahabir RC, Baynosa RC, Khiabani KT. Levels of evidence in research published in The Journal of Bone and Joint Surgery (American Volume) over the last thirty years. J Bone Joint Surg Am. 2009;91:425–428.PubMedCrossRef

12.

Keith MW, Masear V, Amadio PC, Andary M, Barth RW, Graham B, Chung K, Maupin K, Watters WC 3rd, Haralson RH 3rd, Turkelson CM, Wies JL, McGowan R. Treatment of carpal tunnel syndrome. J Am Acad Orthop Surg. 2009;17:397–405.PubMed

13.

Kuhn JE. Exercise in the treatment of rotator cuff impingement: a systematic review and a synthesized evidence-based rehabilitation protocol. J Shoulder Elbow Surg. 2009;18:138–160.PubMedCrossRef

14.

Lichtman DM, Bindra RR, Boyer MI, Putnam MD, Ring D, Slutsky DJ, Taras JS, Watters WC 3rd, Goldberg MJ, Keith M, Turkelson CM, Wies JL, Haralson RH 3rd, Boyer KM, Hitchcock K, Raymond L. Treatment of distal radius fractures. J Am Acad Orthop Surg. 2010;18:180–189.PubMed

15.

Loiselle F, Mahabir RC, Harrop AR. Levels of evidence in plastic surgery research over 20 years. Plast Reconstr Surg. 2008;121:207e–211e.PubMedCrossRef

16.

NHS National Institute for Health and Clinical Excellence. Methods for development of NICE public health guidance. Available at: http://www.nice.org.uk/nicemedia/pdf/CPHEMethodsManual.pdf. Accessed June 20, 2013.

17.

Obremskey WT, Pappas N, Attallah-Wasif E, Tornetta P 3rd, Bhandari M. Level of evidence in orthopaedic journals. J Bone Joint Surg Am. 2005;87:2632–2638.PubMedCrossRef

18.

Richmond J, Hunter D, Irrgang J, Jones MH, Levy B, Marx R, Snyder-Mackler L, Watters WC 3rd, Haralson RH 3rd, Turkelson CM, Wies JL, Boyer KM, Anderson S, St Andre J, Sluka P, McGowan R; American Academy of Orthopaedic Surgeons. Treatment of osteoarthritis of the knee (nonarthroplasty). J Am Acad Orthop Surg. 2009;17:591–600.

19.

Sackett DL. Rules of evidence and clinical recommendations on the use of antithrombotic agents. Chest. 1986;89(2 suppl):2S–3S.PubMed

20.

Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312:71–72.PubMedCrossRef

21.

Schmidt AH, Zhao G, Turkelson C. Levels of evidence at the AAOS meeting: can authors rate their own submissions, and do other raters agree? J Bone Joint Surg Am. 2009;91:867–873.PubMedCrossRef

22.

Schwartz WB, Gorry GA, Kassirer JP, Essig A. Decision analysis and clinical judgment. Am J Med. 1973;55:459–472.PubMedCrossRef

23.

Thornton J, Alderson P, Tan T, Turner C, Latchem S, Shaw E, Ruiz F, Reken S, Mugglestone MA, Hill J, Neilson J, Westby M, Francis K, Whittington C, Siddiqui F, Sharma T, Kelly V, Ayiku L, Chamberlain K. Introducing GRADE across the NICE clinical guideline program. J Clin Epidemiol. 2013;66:124–131.PubMedCrossRef

24.

Turpen RM, Fesperman SF, Sultan S, Borawski KM, Norris RD, Klink J, Sur RL, Breau RH, Krupski TL, Dahm P. Levels of evidence ratings in the urological literature: an assessment of interobserver agreement. BJU Int. 2010;105:602–606.PubMedCrossRef

25.

Ubbink DT, Legemate DA. Evidence-based surgery. Br J Surg. 2004;91:1091–1092.PubMedCrossRef

26.

US Preventive Services Task Force. Guide to Clinical Preventive Services: Report of the US Preventive Services Task Force. Collingdale, PA: DIANE Publishing; 1989.

27.

Voleti PB, Donegan DJ, Baldwin KD, Lee GC. Level of evidence of presentations at American Academy of Orthopaedic Surgeons annual meetings. J Bone Joint Surg Am. 2012;94:e50.PubMedCrossRef

28.

Voleti PB, Donegan DJ, Kim TWB, Lee GC. Level of evidence: does it change the rate of publication and time to publication of American Academy of Orthopaedic Surgeons presentations? J Bone Joint Surg Am. 2013;95:e2.PubMed

29.

Wasserman JM, Wynn R, Bash TS, Rosenfeld RM. Levels of evidence in otolaryngology journals. Otolaryngol Head Neck Surg. 2006;134:717–723.PubMedCrossRef

30.

Wingfield K, Matheson GO, Meeuwisse WH. Preparticipation evaluation: an evidence-based review. Clin J Sport Med. 2004;14:109–122.PubMedCrossRef

31.

Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85:1–3.PubMedCrossRef

32.

Wupperman R, Davis R, Obremskey WT. Level of evidence in Spine compared to other orthopedic journals. Spine (Phila Pa 1976). 2007;32:388–393.

33.

Xu CC, Côté DW, Chowdhury RH, Morrissey AT, Ansari K. Trends in level of evidence in facial plastic surgery research. Plast Reconstr Surg. 2011;127:1499–1504.PubMedCrossRef

Title: Have Levels of Evidence Improved the Quality of Orthopaedic Research?
Authors: Brian P. Cunningham, MD
Samuel Harmsen, MD
Chris Kweon, MD
Jason Patterson, MD
Robert Waldrop, MD
Alex McLaren, MD
Ryan McLemore, PhD
Publication date: 01-11-2013
Publisher: Springer US
Published in: Clinical Orthopaedics and Related Research® / Issue 11/2013
Print ISSN: 0009-921X
Electronic ISSN: 1528-1132
DOI: https://doi.org/10.1007/s11999-013-3159-4

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Have Levels of Evidence Improved the Quality of Orthopaedic Research?

Abstract

Background

Questions/purposes

Methods

Results

Conclusions

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Questions/purposes

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 11/2013

Stakeholders in Outcome Measures: Review From a Clinical Perspective

Letter to the Editor: Does Vitamin D Improve Osteoarthritis of the Knee: A Randomized Controlled Pilot Trial

Do Monoblock Cups Improve Survivorship, Decrease Wear, or Reduce Osteolysis in Uncemented Total Hip Arthroplasty?

The Oxford Unicompartmental Knee Fails at a High Rate in a High-volume Knee Practice

Prosthesis Failure Within 2 Years of Implantation Is Highly Predictive of Infection

CORR Insights®: Does Vitamin D Improve Osteoarthritis of the Knee: A Randomized Controlled Pilot Trial