Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2013

Open Access 01-12-2013 | Research article

Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: a descriptive analysis

Authors: Susan Armijo-Olivo, Jorge Fuentes, Maria Ospina, Humam Saltaji, Lisa Hartling

Published in: BMC Medical Research Methodology | Issue 1/2013

Login to get access

Abstract

Background

Assessing the risk of bias of randomized controlled trials (RCTs) is crucial to understand how biases affect treatment effect estimates. A number of tools have been developed to evaluate risk of bias of RCTs; however, it is unknown how these tools compare to each other in the items included. The main objective of this study was to describe which individual items are included in RCT quality tools used in general health and physical therapy (PT) research, and how these items compare to those of the Cochrane Risk of Bias (RoB) tool.

Methods

We used comprehensive literature searches and a systematic approach to identify tools that evaluated the methodological quality or risk of bias of RCTs in general health and PT research. We extracted individual items from all quality tools. We calculated the frequency of quality items used across tools and compared them to those in the RoB tool. Comparisons were made between general health and PT quality tools using Chi-squared tests.

Results

In addition to the RoB tool, 26 quality tools were identified, with 19 being used in general health and seven in PT research. The total number of quality items included in general health research tools was 130, compared with 48 items across PT tools and seven items in the RoB tool. The most frequently included items in general health research tools (14/19, 74%) were inclusion and exclusion criteria, and appropriate statistical analysis. In contrast, the most frequent items included in PT tools (86%, 6/7) were: baseline comparability, blinding of investigator/assessor, and use of intention-to-treat analysis. Key items of the RoB tool (sequence generation and allocation concealment) were included in 71% (5/7) of PT tools, and 63% (12/19) and 37% (7/19) of general health research tools, respectively.

Conclusions

There is extensive item variation across tools that evaluate the risk of bias of RCTs in health research. Results call for an in-depth analysis of items that should be used to assess risk of bias of RCTs. Further empirical evidence on the use of individual items and the psychometric properties of risk of bias tools is needed.
Appendix
Available only for authorised users
Literature
1.
Khan K, Ter Riet G, Popay J, Nixon J, Kleijnen J: Satge II. Conducting the Review. Phase 5 Study Quality Assessment. Undertaking Systematic Reviews of research Effectiveness CRD’s Guidance for those carrying out or commissioning reviews. 2001, York: Centre for Reviews and Dissemination, 1-20.
2.
Verhagen AP, de Vet HC, de Bie RA, Boers M, van den Brandt PA: The art of quality assessment of RCTs included in systematic reviews. J Clin Epidemiol. 2001, 54 (7): 651-654. 10.1016/S0895-4356(00)00360-7.CrossRefPubMed
3.
Juni P, Altman DG, Egger M: Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001, 323 (7303): 42-46. 10.1136/bmj.323.7303.42.CrossRefPubMedPubMedCentral
4.
Higgins JPT, Altman DG, Goetzsche PC, Juni P, Moher D, Oxman AD, Savovicć J, Schulz KF, Weeks L, Sterne JAC: The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011, 343 (7829): d5928-10.1136/bmj.d5928CrossRefPubMedPubMedCentral
5.
Armijo-Olivo S, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ: Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther. 2008, 88 (2): 156-175. 10.2522/ptj.20070147.CrossRef
6.
Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S: Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995, 16 (1): 62-73. 10.1016/0197-2456(94)00031-W.CrossRefPubMed
7.
Colle F, Rannou F, Revel M, Fermanian J, Poiraudeau S: Impact of quality scales on levels of evidence inferred from a systematic review of exercise therapy and low back pain. Arch Phys Med Rehabil. 2002, 83 (12): 1745-1752. 10.1053/apmr.2002.35657.CrossRefPubMed
8.
Herbison P, Hay-Smith J, Gillespie WJ: Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol. 2006, 59 (12): 1249-1256.CrossRefPubMed
9.
Armijo-Olivo S, Stiles C, Hagen N, Biondo P, Cummings G: Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias tool and the Effectve Public Health Practice Project Quality Assessment Tool: methodological research. J Eval Clin Pract. 2012, 18 (1): 12-18. 10.1111/j.1365-2753.2010.01516.x.CrossRefPubMed
10.
Higgins J, Altman D: Chapter 8: Assessing risk of bias in included studies. Cochrane Handbook for Systematic Reviews of Interventions version 50. Edited by: Higgins J, Green S. 2008, Chichester, UK: John Wiley & Sons, LtdCrossRef
11.
Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC: Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007, 36 (4): 847-857. 10.1093/ije/dym087.CrossRefPubMed
12.
Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJG, Sterne JAC: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. BMJ. 2008, 336 (7644): 601-605. 10.1136/bmj.39465.451748.AD.CrossRefPubMedPubMedCentral
13.
Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Seida JK, Klassen TP: Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009, 339 (7728): 1017-
14.
Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet. 1998, 352 (9128): 609-613. 10.1016/S0140-6736(98)01085-X.CrossRefPubMed
15.
Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273 (5): 408-412. 10.1001/jama.1995.03520290060030.CrossRefPubMed
16.
17.
Delgado-Rodriguez M, Llorca J: Bias. J Epidemiol Commun Health. 2004, 58 (8): 635-641. 10.1136/jech.2003.008466.CrossRef
18.
Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA: A systematic review of the content of critical appraisal tools. BMC Med Res Methodol. 2004, 4 (1): 22-10.1186/1471-2288-4-22.CrossRefPubMedPubMedCentral
19.
Dechartres A, Charles P, Hopewell S, Ravaud P, Altman DG: Reviews assessing the quality or the reporting of randomized controlled trials are increasing over time but raised questions about how quality is assessed. J Clin Epidemiol. 2011, 64 (2): 136-144. 10.1016/j.jclinepi.2010.04.015.CrossRefPubMed
20.
Kunz R, Autti-Ramo I, Anttila H, Malmivaara A, Makela M: A systematic review finds that methodological quality is better than its reputation but can be improved in physiotherapy trials in childhood cerebral palsy. J Clin Epidemiol. 2006, 59 (12): 1239-1248.CrossRefPubMed
21.
Streiner D, Norman G: Validity. Health Measurements Scales. Edited by: Streiner D, Norman G. 2004, Oxford: Oxford University Press, 172-193.
22.
Streiner D, Norman G: Reliability. Health Measurements Scales. Edited by: Streiner D, Norman G. 2004, Oxford: Oxford University Press, 126-152.
23.
Streiner D, Norman G: Measuring Change. Health Measurements Scales. Edited by: Streiner D, Norman G. 2004, Oxford: Oxford University Press, 194-212.
24.
Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, Bouter LM, de Vet HCW: Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007, 60 (1): 34-42. 10.1016/j.jclinepi.2006.03.012.CrossRefPubMed
25.
Whiting P, Rutjes AWS, Reitsma JB, Glas AS, Bossuyt PMM, Kleijnen J: Sources of variation and bias in studies of diagnostic accuracy: a systematic review. An Intern Med. 2004, 140 (3): 189-202. 10.7326/0003-4819-140-3-200402030-00010.CrossRef
26.
27.
Viswanathan M, Berkman ND: Development of the RTI item bank on risk of bias and precision of observational studies. J Clin Epidemiol. 2011, 65 (2): 163-178.CrossRefPubMed
28.
Hayden JA, Cotte P, Bombardier C: Evaluation of the quality of prognosis studies in systematic reviews. An Intern Med. 2006, 144 (6): 427-437. 10.7326/0003-4819-144-6-200603210-00010.CrossRef
29.
Lash TL, Fox MP, Fink AK: Multiple Bias Modeling: Applying Quantitatvie Bias Analysis to. 2009, Springer New York: Observational Epidemiologic ResearchCrossRef
30.
Byrt T: How good is that agreement?. Epidemiology (Cambridge, Mass). 1996, 7 (5): 561-
31.
Jurgens T, Whelan AM, MacDonald M, Lord L: Development and evaluation of an instrument for the critical appraisal of randomized controlled trials of natural products. BMC Complem Altern Med. 2009, 9: 11-10.1186/1472-6882-9-11. doi:10.1186/1472-6882-9-11CrossRef
32.
Gerber AJ, Kocsis JH, Milrod BL, Roose SP, Barber JP, Thase ME, Perkins P, Leon AC: A quality-based review of randomized controlled trials of psychodynamic psychotherapy. Am J Psychiatr. 2011, 168 (1): 19-28. 10.1176/appi.ajp.2010.08060843.CrossRefPubMed
33.
Kocsis JH, Gerber AJ, Milrod B, Roose SP, Barber J, Thase ME, Perkins P, Leon AC: A new scale for assessing the quality of randomized clinical trials of psychotherapy. Compr Psychiatr. 2010, 51 (3): 319-324. 10.1016/j.comppsych.2009.07.001.CrossRef
34.
Cipriani A, Malvini L, Furukawa TA, Barbui C: Relationship between quality of reports of antidepressant randomized controlled trials and treatment estimates: Systematic review, meta-analysis, and meta-regression analysis. J Clin Psychopharmacol. 2007, 27 (4): 352-356. 10.1097/jcp.0b013e3180a9081f.CrossRefPubMed
35.
Moncrieff J, Churchill R, Colin Drummond D, McGuire H: Development of a quality assessment instrument for trials of treatments for depression and neurosis. Int J Meth Psychiatr Res. 2001, 10 (3): 126-133. 10.1002/mpr.108.CrossRef
36.
Boutron I, Ravaud P, Moher D, Tugwell P, Giraudeau B, Poiraudeau S, Nizard R: A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus. J Clin Epidemiol. 2005, 58 (12): 1233-1240. 10.1016/j.jclinepi.2005.05.004.CrossRefPubMed
37.
Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomized clinical trials: Is blinding necessary?. Control Clin Trials. 1996, 17 (1): 1-12. 10.1016/0197-2456(95)00134-4.CrossRefPubMed
38.
De Vet HCW, De Bie RA, Van Der Heijden GJMG, Verhagen AP, Sijpkes P, Knipschild PG: Systematic reviews on the basis of methodological criteria. Physiotherapy. 1997, 83 (6): 284-289. 10.1016/S0031-9406(05)66175-5.CrossRef
39.
Verhagen AP, de Vet HC, de Bie RA, Kessels AG, Boers M, Bouter LM, Knipschild PG: The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol. 1998, 51 (12): 1235-1241. 10.1016/S0895-4356(98)00131-0.CrossRefPubMed
40.
Sherrington C, Herbert RD, Maher CG, Moseley AM: PEDro: A database of randomized trials and systematic reviews in physiotherapy. Man Ther. 2000, 5 (4): 223-226. 10.1054/math.2000.0372.CrossRefPubMed
41.
Moseley AM, Herbert RD, Sherrington C, Maher CG: Evidence for physiotherapy practice: a survey of the Physiotherapy Evidence Database (PEDro). Aust J Physiother. 2002, 48 (1): 43-49.CrossRefPubMed
42.
Van Tulder MW, Assendelft WJJ, Koes BW, Bouter LM, Bombardier C, Nachemson AL, Esmail R, Deyo RA, Shekelle PG, Bouter LM, et al: Method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group for spinal disorders. Spine. 1997, 22 (20): 2323-2330. 10.1097/00007632-199710150-00001.CrossRefPubMed
43.
Van Tulder M, Furlan A, Bombardier C, Bouter L: Updated method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group. Spine. 2003, 28 (12): 1290-1299.PubMed
44.
Bizzini M, Childs JD, Piva SR: Systematic review of the quality of randomized controlled trials for patellofemoral pain syndrome. J Orthop Sports Phys Ther. 2003, 33 (1): 4-20. 10.2519/jospt.2003.33.1.4.CrossRefPubMed
45.
Chalmers TC, Smith H, Blackburn B: A method for assessing the quality of a randomized control trial. Control Clin Trials. 1981, 2 (1): 31-49. 10.1016/0197-2456(81)90056-8.CrossRefPubMed
46.
Reisch JS, Tyson JE, Mize SG: Aid to the evaluation of therapeutic studies. Pediatrics. 1989, 84 (5): 815-827.PubMed
47.
Andrew E: Method for assessment of the reporting standard of clinical trials with roentgen contrast media. Acta Radiologica - Series Diagnosis. 1984, 25 (1): 55-58.CrossRef
48.
Imperiale TF, McCullough AJ: Do corticosteroids reduce mortality from alcoholic hepatitis? A meta-analysis of the randomized trials. Ann Intern Med. 1990, 113 (4): 299-307. 10.7326/0003-4819-113-4-299.CrossRefPubMed
49.
Detsky AS, Naylor CD, O’Rourke K, McGeer AJ, L’Abbe KA: Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol. 1992, 45 (3): 255-265. 10.1016/0895-4356(92)90085-2.CrossRefPubMed
50.
Cho MK, Bero LA: Instruments for assessing the quality of drug studies published in the medical literature. JAMA. 1994, 272 (2): 101-104. 10.1001/jama.1994.03520020027007.CrossRefPubMed
51.
Balas EA, Austin SM, Ewigman BG, Brown GD, Mitchell JA: Methods of randomized controlled clinical trials in health services research. Med Care. 1995, 33 (7): 687-299. 10.1097/00005650-199507000-00005.CrossRefPubMed
52.
Sindhu F, Carpenter L, Seers K: Development of a tool to rate the quality assessment of randomized controlled trials using a Delphi technique. J Adv Nurs. 1997, 25 (6): 1262-1268. 10.1046/j.1365-2648.1997.19970251262.x.CrossRefPubMed
53.
Downs SH, Black N: The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Commun Health. 1998, 52 (6): 377-384. 10.1136/jech.52.6.377.CrossRef
54.
Nguyen QV, Bezemer PD, Habets L, Prahl-Andersen B: A systematic review of the relationship between overjet size and traumatic dental injuries. Eur J Orthodont. 1999, 21 (5): 503-515. 10.1093/ejo/21.5.503.CrossRef
55.
Smith LA, Oldman AD, McQuay HJ, Moore RA: Teasing apart quality and validity in systematic reviews: an example from acupuncture trials in chronic neck and back pain. Pain. 2000, 86 (1–2): 119-132.CrossRefPubMed
56.
Arrive L, Renard R, Carrat F, Belkacem A, Dahan H, Le Hir P, Monnier-Cholley L, Tubiana JM: A scale of methodological quality for clinical studies of radiologic examinations. Radiology. 2000, 217 (1): 69-74.CrossRefPubMed
57.
Huwiler-Muntener K, Juni P, Junker C, Egger M: Quality of reporting of randomized trials as a measure of methodologic quality. JAMA. 2002, 287 (21): 2801-2804. 10.1001/jama.287.21.2801.CrossRefPubMed
58.
Yates SL, Morley S, Eccleston C, Williams ACDC: A scale for rating the quality of psychological trials for pain. Pain. 2005, 117 (3): 314-325. 10.1016/j.pain.2005.06.018.CrossRefPubMed
59.
Graham N, Haines T, Goldsmith CH, Gross A, Burnie S, Shahzad U, Talovikova E: Reliability of three assessment tools used to evaluate randomized controlled trials for treatment of neck pain. Spine. 2012, 37 (6): 515-522. 10.1097/BRS.0b013e31822671eb.CrossRefPubMed
60.
Hartling L, Bond K, Vandermeer B, Seida J, Dryden D, Rowe B: Applying the Risk of Bias tool in a systematic review of combination longacting betaagonists and inhaled corticosteroids for persistent asthma. PLoS Med. 2011, 6 (2): e17242-CrossRef
61.
Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG: Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research. J Eval Clin Pract. 2010, 1-7.
62.
Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, Hempel S, Shekelle P, Dryden DM: Testing the Risk of Bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2012, S0895-4356(12)00217-X. doi: 10.1016/j.jclinepi.2012.07.005
63.
Juni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282 (11): 1054-1060. 10.1001/jama.282.11.1054.CrossRefPubMed
64.
Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, Djulbegovic B: Bad reporting does not mean bad methods for randomised trials: Observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ. 2004, 328 (7430): 22-24. 10.1136/bmj.328.7430.22.CrossRefPubMedPubMedCentral
65.
Sanderson S, Tatt ID, Higgins JPT: Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007, 36 (3): 666-676. 10.1093/ije/dym018.CrossRefPubMed
Metadata
Title
Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: a descriptive analysis
Authors
Susan Armijo-Olivo
Jorge Fuentes
Maria Ospina
Humam Saltaji
Lisa Hartling
Publication date
01-12-2013
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2013
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-13-116

Other articles of this Issue 1/2013

BMC Medical Research Methodology 1/2013 Go to the issue

Research article

Bayesian methods to determine performance differences and to quantify variability among centers in multi-center trials: the IHAST trial

Research article

Ensemble-based methods for forecasting census in hospital units

Research article

Validity of self-reported height and weight among adolescents: the importance of reporting capability

Research article

Participant recruitment in sensitive surveys: a comparative trial of ‘opt in’ versus ‘opt out’ approaches

Research article

Predicting hemoglobin levels in whole blood donors using transition models and mixed effects models

Research article

The use of systematic reviews in the planning, design and conduct of randomised trials: a retrospective cohort of NIHR HTA funded trials