Skip to main content
Top
Published in: European Journal of Trauma and Emergency Surgery 6/2022

Open Access 09-07-2022 | Original Article

How to assess applicability and methodological quality of comparative studies of operative interventions in orthopedic trauma surgery

Authors: Kim Luijken, Bryan J. M. van de Wall, Lotty Hooft, Luke P. H. Leenen, R. Marijn Houwert, Rolf H. H. Groenwold, on behalf of the NEXT Study Group

Published in: European Journal of Trauma and Emergency Surgery | Issue 6/2022

Login to get access

Abstract

Purpose

It is challenging to generate and subsequently implement high-quality evidence in surgical practice. A first step would be to grade the strengths and weaknesses of surgical evidence and appraise risk of bias and applicability. Here, we described items that are common to different risk-of-bias tools. We explained how these could be used to assess comparative operative intervention studies in orthopedic trauma surgery, and how these relate to applicability of results.

Methods

We extracted information from the Cochrane risk-of-bias-2 (RoB-2) tool, Risk Of Bias In Non-randomised Studies—of Interventions tool (ROBINS-I), and Methodological Index for Non-Randomized Studies (MINORS) criteria and derived a concisely formulated set of items with signaling questions tailored to operative interventions in orthopedic trauma surgery.

Results

The established set contained nine items: population, intervention, comparator, outcome, confounding, missing data and selection bias, intervention status, outcome assessment, and pre-specification of analysis. Each item can be assessed using signaling questions and was explained using good practice examples of operative intervention studies in orthopedic trauma surgery.

Conclusion

The set of items will be useful to form a first judgment on studies, for example when including them in a systematic review. Existing risk of bias tools can be used for further evaluation of methodological quality. Additionally, the proposed set of items and signaling questions might be a helpful starting point for peer reviewers and clinical readers.
Appendix
Available only for authorised users
Literature
1.
go back to reference Robinson A, et al. The challenges of translating the results of randomized controlled trials in orthopaedic surgery into clinical practice. Bone Joint J. 2019;101(B2):121–3.PubMedCrossRef Robinson A, et al. The challenges of translating the results of randomized controlled trials in orthopaedic surgery into clinical practice. Bone Joint J. 2019;101(B2):121–3.PubMedCrossRef
2.
go back to reference Axelrod D, et al. The Canadian orthopaedic trauma society: lessons learned from 30 years of collaborative, high-impact research in fracture care. Bone Joint J. 2021;103(5):898–901.PubMedCrossRef Axelrod D, et al. The Canadian orthopaedic trauma society: lessons learned from 30 years of collaborative, high-impact research in fracture care. Bone Joint J. 2021;103(5):898–901.PubMedCrossRef
3.
go back to reference Oberkofler CE, et al. Procedural surgical RCTs in daily practice: do surgeons adopt or is it just a waste of time? Ann Surg. 2019;270(5):727–34.PubMedCrossRef Oberkofler CE, et al. Procedural surgical RCTs in daily practice: do surgeons adopt or is it just a waste of time? Ann Surg. 2019;270(5):727–34.PubMedCrossRef
4.
go back to reference Chapman SJ, et al. Research waste in surgical randomized controlled trials. J Br Surg. 2019;106(11):1464–71.CrossRef Chapman SJ, et al. Research waste in surgical randomized controlled trials. J Br Surg. 2019;106(11):1464–71.CrossRef
5.
go back to reference Houwert RM, et al. A reaction to the editorial “meta-analyses and systematic reviews: JBJS policy revisited.” J Bone Joint Surg. 2021;103(10):849.CrossRef Houwert RM, et al. A reaction to the editorial “meta-analyses and systematic reviews: JBJS policy revisited.” J Bone Joint Surg. 2021;103(10):849.CrossRef
6.
go back to reference Beks RB, et al. When observational studies are as helpful as randomized trials: examples from orthopedic trauma. J Trauma Acute Care Surg. 2019;87(3):730–2.PubMedCrossRef Beks RB, et al. When observational studies are as helpful as randomized trials: examples from orthopedic trauma. J Trauma Acute Care Surg. 2019;87(3):730–2.PubMedCrossRef
7.
go back to reference Sanderson S, Tatt ID, Higgins J. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007;36(3):666–76.PubMedCrossRef Sanderson S, Tatt ID, Higgins J. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007;36(3):666–76.PubMedCrossRef
8.
go back to reference Katrak P, et al. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol. 2004;4(1):1–11.CrossRef Katrak P, et al. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol. 2004;4(1):1–11.CrossRef
9.
go back to reference Moher D, et al. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995;16(1):62–73.PubMedCrossRef Moher D, et al. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995;16(1):62–73.PubMedCrossRef
10.
go back to reference D’Andrea E, et al. How well can we assess the validity of non-randomised studies of medications? a systematic review of assessment tools. BMJ Open. 2021;11(3): e043961.PubMedPubMedCentralCrossRef D’Andrea E, et al. How well can we assess the validity of non-randomised studies of medications? a systematic review of assessment tools. BMJ Open. 2021;11(3): e043961.PubMedPubMedCentralCrossRef
13.
go back to reference Guyatt GH, et al. GRADE guidelines: Framing the question and deciding on important outcomes. J Clin Epidemiol. 2011;64(4):395–400.PubMedCrossRef Guyatt GH, et al. GRADE guidelines: Framing the question and deciding on important outcomes. J Clin Epidemiol. 2011;64(4):395–400.PubMedCrossRef
14.
go back to reference Fjalestad T, et al. Surgical treatment with an angular stable plate for complex displaced proximal humeral fractures in elderly patients: a randomized controlled trial. J Orthop Trauma. 2012;26(2):98–106.PubMedCrossRef Fjalestad T, et al. Surgical treatment with an angular stable plate for complex displaced proximal humeral fractures in elderly patients: a randomized controlled trial. J Orthop Trauma. 2012;26(2):98–106.PubMedCrossRef
15.
go back to reference Knobe M, et al. Is helical blade nailing superior to locked minimally invasive plating in unstable pertrochanteric fractures? Clin Orthop Relat Res®. 2012;470(8):2302–12.PubMedCrossRef Knobe M, et al. Is helical blade nailing superior to locked minimally invasive plating in unstable pertrochanteric fractures? Clin Orthop Relat Res®. 2012;470(8):2302–12.PubMedCrossRef
16.
go back to reference Constant C, Murley A. A clinical method of functional assessment of the shoulder. Clin Orthop Relat Res. 1987;214:160–4.CrossRef Constant C, Murley A. A clinical method of functional assessment of the shoulder. Clin Orthop Relat Res. 1987;214:160–4.CrossRef
17.
go back to reference Rangan A, et al. Surgical vs nonsurgical treatment of adults with displaced fractures of the proximal humerus: the PROFHER randomized clinical trial. JAMA. 2015;313(10):1037–47.PubMedCrossRef Rangan A, et al. Surgical vs nonsurgical treatment of adults with displaced fractures of the proximal humerus: the PROFHER randomized clinical trial. JAMA. 2015;313(10):1037–47.PubMedCrossRef
19.
go back to reference Hernán MA, Robins JM. Causal inference: what if. Boca Raton: Chapman & Hall/CRC; 2020. Hernán MA, Robins JM. Causal inference: what if. Boca Raton: Chapman & Hall/CRC; 2020.
20.
go back to reference Beks RB, et al. Rib fixation versus non-operative treatment for flail chest and multiple rib fractures after blunt thoracic trauma: a multicenter cohort study. Eur J Trauma Emerg Surg. 2019;45(4):655–63.PubMedCrossRef Beks RB, et al. Rib fixation versus non-operative treatment for flail chest and multiple rib fractures after blunt thoracic trauma: a multicenter cohort study. Eur J Trauma Emerg Surg. 2019;45(4):655–63.PubMedCrossRef
22.
go back to reference Jenkinson RJ, et al. Delayed wound closure increases deep-infection rate associated with lower-grade open fractures: a propensity-matched cohort study. JBJS. 2014;96(5):380–6.CrossRef Jenkinson RJ, et al. Delayed wound closure increases deep-infection rate associated with lower-grade open fractures: a propensity-matched cohort study. JBJS. 2014;96(5):380–6.CrossRef
24.
go back to reference VanderWeele TJ. On the relative nature of overadjustment and unnecessary adjustment. Epidemiology. 2009;20(4):496–9.PubMedCrossRef VanderWeele TJ. On the relative nature of overadjustment and unnecessary adjustment. Epidemiology. 2009;20(4):496–9.PubMedCrossRef
25.
26.
go back to reference Sun G-W, Shook TL, Kay GL. Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol. 1996;49(8):907–16.PubMedCrossRef Sun G-W, Shook TL, Kay GL. Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol. 1996;49(8):907–16.PubMedCrossRef
27.
29.
31.
go back to reference Lee KJ, et al. Framework for the treatment and reporting of missing data in observational studies: the treatment and reporting of missing data in observational studies framework. J Clin Epidemiol. 2021;134:79–88.PubMedPubMedCentralCrossRef Lee KJ, et al. Framework for the treatment and reporting of missing data in observational studies: the treatment and reporting of missing data in observational studies framework. J Clin Epidemiol. 2021;134:79–88.PubMedPubMedCentralCrossRef
32.
33.
go back to reference Carpenter JR, Smuk M. Missing data: A statistical framework for practice. Biom J. 2021;63(5):915–47.PubMedCrossRef Carpenter JR, Smuk M. Missing data: A statistical framework for practice. Biom J. 2021;63(5):915–47.PubMedCrossRef
34.
go back to reference Groenwold RH, Dekkers OM. Missing data: the impact of what is not there. Eur J Endocrinol. 2020;183(4):E7–9.PubMedCrossRef Groenwold RH, Dekkers OM. Missing data: the impact of what is not there. Eur J Endocrinol. 2020;183(4):E7–9.PubMedCrossRef
36.
go back to reference Van der Meijden OA, et al. Operative treatment of dislocated midshaft clavicular fractures: plate or intramedullary nail fixation?: a randomized controlled trial. JBJS. 2015;97(8):613–9.CrossRef Van der Meijden OA, et al. Operative treatment of dislocated midshaft clavicular fractures: plate or intramedullary nail fixation?: a randomized controlled trial. JBJS. 2015;97(8):613–9.CrossRef
37.
go back to reference Scholtes VA, Terwee CB, Poolman RW. What makes a measurement instrument valid and reliable? Injury. 2011;42(3):236–40.PubMedCrossRef Scholtes VA, Terwee CB, Poolman RW. What makes a measurement instrument valid and reliable? Injury. 2011;42(3):236–40.PubMedCrossRef
38.
39.
go back to reference Ochen Y, et al. Surgical treatment of Neer type II and type V lateral clavicular fractures: comparison of hook plate versus superior plate with lateral extension: a retrospective cohort study. Eur J Orthop Surg Traumatol. 2019;29(5):989–97.PubMedPubMedCentralCrossRef Ochen Y, et al. Surgical treatment of Neer type II and type V lateral clavicular fractures: comparison of hook plate versus superior plate with lateral extension: a retrospective cohort study. Eur J Orthop Surg Traumatol. 2019;29(5):989–97.PubMedPubMedCentralCrossRef
40.
go back to reference Nauth A, et al. Fracture fixation in the operative management of hip fractures (FAITH): an international, multicentre, randomised controlled trial. Lancet. 2017;389(10078):1519–27.CrossRef Nauth A, et al. Fracture fixation in the operative management of hip fractures (FAITH): an international, multicentre, randomised controlled trial. Lancet. 2017;389(10078):1519–27.CrossRef
41.
go back to reference Groenwold RH, et al. Multiple testing: when is many too much? Eur J of Endocrinol. 2021;184(3):E11–4.CrossRef Groenwold RH, et al. Multiple testing: when is many too much? Eur J of Endocrinol. 2021;184(3):E11–4.CrossRef
42.
go back to reference Gamble C, et al. Guidelines for the content of statistical analysis plans in clinical trials. JAMA. 2017;318(23):2337–43.PubMedCrossRef Gamble C, et al. Guidelines for the content of statistical analysis plans in clinical trials. JAMA. 2017;318(23):2337–43.PubMedCrossRef
44.
45.
go back to reference Smeeing DPJ, et al. Weight-bearing or non-weight-bearing after surgical treatment of ankle fractures: a multicenter randomized controlled trial. Eur J Trauma Emerg Surg. 2020;46(1):121–30.PubMedCrossRef Smeeing DPJ, et al. Weight-bearing or non-weight-bearing after surgical treatment of ankle fractures: a multicenter randomized controlled trial. Eur J Trauma Emerg Surg. 2020;46(1):121–30.PubMedCrossRef
46.
go back to reference Briet JP, et al. Weight bearing or non-weight bearing after surgically fixed ankle fractures, the WOW! study: study protocol for a randomized controlled trial. Trials. 2015;16(1):1–8.CrossRef Briet JP, et al. Weight bearing or non-weight bearing after surgically fixed ankle fractures, the WOW! study: study protocol for a randomized controlled trial. Trials. 2015;16(1):1–8.CrossRef
47.
49.
go back to reference McCulloch P, et al. No surgical innovation without evaluation: the IDEAL recommendations. Lancet. 2009;374(9695):1105–12.PubMedCrossRef McCulloch P, et al. No surgical innovation without evaluation: the IDEAL recommendations. Lancet. 2009;374(9695):1105–12.PubMedCrossRef
50.
go back to reference Khachane A, et al. Appraising the uptake and use of the IDEAL framework and recommendations: a review of the literature. Int J Surg. 2018;57:84–90.PubMedCrossRef Khachane A, et al. Appraising the uptake and use of the IDEAL framework and recommendations: a review of the literature. Int J Surg. 2018;57:84–90.PubMedCrossRef
51.
go back to reference Bilbro NA, et al. The ideal reporting guidelines: a Delphi consensus statement stage specific recommendations for reporting the evaluation of surgical innovation. Ann Surg. 2021;273(1):82–5.PubMedCrossRef Bilbro NA, et al. The ideal reporting guidelines: a Delphi consensus statement stage specific recommendations for reporting the evaluation of surgical innovation. Ann Surg. 2021;273(1):82–5.PubMedCrossRef
53.
go back to reference Guyatt GH, et al. GRADE guidelines: a new series of articles in the journal of clinical epidemiology. J Clin Epidemiol. 2011;64(4):380–2.PubMedCrossRef Guyatt GH, et al. GRADE guidelines: a new series of articles in the journal of clinical epidemiology. J Clin Epidemiol. 2011;64(4):380–2.PubMedCrossRef
55.
go back to reference Murad MH, et al. New evidence pyramid. BMJ Evid-Based Med. 2016;21(4):125–7.CrossRef Murad MH, et al. New evidence pyramid. BMJ Evid-Based Med. 2016;21(4):125–7.CrossRef
56.
go back to reference Jeyaraman MM, et al. Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden. J Clin Epidemiol. 2020;128:140–7.PubMedCrossRef Jeyaraman MM, et al. Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden. J Clin Epidemiol. 2020;128:140–7.PubMedCrossRef
57.
go back to reference Minozzi S, et al. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application. J Clin Epidemiol. 2020;126:37–44.PubMedCrossRef Minozzi S, et al. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application. J Clin Epidemiol. 2020;126:37–44.PubMedCrossRef
58.
go back to reference Minozzi S, et al. Risk of bias in nonrandomized studies of interventions showed low inter-rater reliability and challenges in its application. J Clin Epidemiol. 2019;112:28–35.PubMedCrossRef Minozzi S, et al. Risk of bias in nonrandomized studies of interventions showed low inter-rater reliability and challenges in its application. J Clin Epidemiol. 2019;112:28–35.PubMedCrossRef
59.
go back to reference Meakins JL. Evidence-based practice: new techniques and technology Canadian journal of surgery. J Can de Chir. 2001;44(4):247–9. Meakins JL. Evidence-based practice: new techniques and technology Canadian journal of surgery. J Can de Chir. 2001;44(4):247–9.
61.
go back to reference VanderWeele TJ. On well-defined hypothetical interventions in the potential outcomes framework. Epidemiology(Cambridge, Mass). 2018;29(4):e24.PubMed VanderWeele TJ. On well-defined hypothetical interventions in the potential outcomes framework. Epidemiology(Cambridge, Mass). 2018;29(4):e24.PubMed
Metadata
Title
How to assess applicability and methodological quality of comparative studies of operative interventions in orthopedic trauma surgery
Authors
Kim Luijken
Bryan J. M. van de Wall
Lotty Hooft
Luke P. H. Leenen
R. Marijn Houwert
Rolf H. H. Groenwold
on behalf of the NEXT Study Group
Publication date
09-07-2022
Publisher
Springer Berlin Heidelberg
Published in
European Journal of Trauma and Emergency Surgery / Issue 6/2022
Print ISSN: 1863-9933
Electronic ISSN: 1863-9941
DOI
https://doi.org/10.1007/s00068-022-02031-9

Other articles of this Issue 6/2022

European Journal of Trauma and Emergency Surgery 6/2022 Go to the issue