Skip to main content
Top
Published in: Systematic Reviews 1/2013

Open Access 01-12-2013 | Protocol

How should we evaluate the risk of bias of physical therapy trials?: a psychometric and meta-epidemiological approach towards developing guidelines for the design, conduct, and reporting of RCTs in Physical Therapy (PT) area: a study protocol

Authors: Susan Armijo-Olivo, Jorge Fuentes, Todd Rogers, Lisa Hartling, Humam Saltaji, Greta G Cummings

Published in: Systematic Reviews | Issue 1/2013

Login to get access

Abstract

Background

Numerous tools and items have been developed in all health areas to assess the risk of bias of randomized controlled trials (RCTs). The Cochrane Collaboration (CC) released a new tool to assess bias in RCTs, based on empirical evidence quantifying the association between some design features and estimates of treatment effects (TEs). However, this evidence is limited to medicine and investigating a selected set of components. No such studies have been conducted in other health areas such as Physical Therapy (PT) and allied health professions. Evidence specific to the PT area is needed to understand and quantify the association between design features and TE estimates to inform practice and decision-making in this field. The overall goal of this project is to provide direction for the design, conduct, reporting and bias assessment of PT RCTs. We will achieve this through the following specific objectives and methods.

Methods/Design

1) to measure the association between methodological components and other factors (for example, PT area, type of intervention, type of outcomes) and TE estimates in RCTs in PT, 40 randomly selected meta-analyses of RCTs involving PT interventions will be identified from the Cochrane Database of Systematic Reviews. Trials will be evaluated independently by two reviewers using the most commonly used tools in the PT field. A two-level analysis will be conducted using a meta-meta-analytic approach; 2) to identify relevant items to evaluate risk of bias of PT trials, an exploratory factor analysis (EFA) will be used to identify the latent structure of the items; 3) to develop guidelines for the design, conduct, reporting, and risk of bias assessment of PT RCTs, items obtained from the factor analysis and the meta-epidemiological approach will be further evaluated by experts in PT through a web-based survey following a Delphi procedure.

Discussion

The results of this project will have a direct impact on research and practice in PT and are valuable to a number of stakeholders: researchers when designing, conducting, and reporting trials; systematic reviewers and meta-analysts when synthesizing trial results; physiotherapists when making day-to-day treatment decision; and, other healthcare decision-makers, such as those developing policy or practice guidelines.
Literature
1.
go back to reference Higgins JPT, Altman DG, Goetzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JAC: The Cochrane collaboration’s tool for assessing risk of bias in randomized trials. BMJ. 2011, 343: 7829-OnlineCrossRef Higgins JPT, Altman DG, Goetzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JAC: The Cochrane collaboration’s tool for assessing risk of bias in randomized trials. BMJ. 2011, 343: 7829-OnlineCrossRef
2.
go back to reference Khan K, Ter-Riet G, Popay J, Nixon J, Kleijnen J: Satge ii. Conducting the review. Phase 5 study quality assessment. Undertaking systematic reviews of research effectiveness CRD’s guidance for those carrying out or commissioning reviews. 2001, York: Center for Reviews and Dissemination, 1-20. Khan K, Ter-Riet G, Popay J, Nixon J, Kleijnen J: Satge ii. Conducting the review. Phase 5 study quality assessment. Undertaking systematic reviews of research effectiveness CRD’s guidance for those carrying out or commissioning reviews. 2001, York: Center for Reviews and Dissemination, 1-20.
3.
go back to reference Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC: Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007, 36 (4): 847-857. 10.1093/ije/dym087.CrossRefPubMed Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC: Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007, 36 (4): 847-857. 10.1093/ije/dym087.CrossRefPubMed
4.
go back to reference Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJG, Sterne JAC: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008, 336 (7644): 601-605. 10.1136/bmj.39465.451748.AD.CrossRefPubMedPubMedCentral Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJG, Sterne JAC: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008, 336 (7644): 601-605. 10.1136/bmj.39465.451748.AD.CrossRefPubMedPubMedCentral
5.
go back to reference Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Seida JK, Klassen TP: Risk of bias versus quality assessment of randomized controlled trials: cross sectional study. BMJ. 2009, 339 (7728): 1017- Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Seida JK, Klassen TP: Risk of bias versus quality assessment of randomized controlled trials: cross sectional study. BMJ. 2009, 339 (7728): 1017-
6.
go back to reference Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP: Does quality of reports of randomized trials affect estimates of intervention efficacy reported in meta-analyses?(see comment). Lancet. 1998, 352 (9128): 609-613. 10.1016/S0140-6736(98)01085-X.CrossRefPubMed Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP: Does quality of reports of randomized trials affect estimates of intervention efficacy reported in meta-analyses?(see comment). Lancet. 1998, 352 (9128): 609-613. 10.1016/S0140-6736(98)01085-X.CrossRefPubMed
7.
go back to reference Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. J Am Med Assoc. 1995, 273 (5): 408-412. 10.1001/jama.1995.03520290060030.CrossRef Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. J Am Med Assoc. 1995, 273 (5): 408-412. 10.1001/jama.1995.03520290060030.CrossRef
8.
go back to reference Berger VW, Weinstein S: Ensuring the comparability of comparison groups: is randomization enough?. Control Clin Trials. 2004, 25 (5): 515-524. 10.1016/j.cct.2004.04.001.CrossRefPubMed Berger VW, Weinstein S: Ensuring the comparability of comparison groups: is randomization enough?. Control Clin Trials. 2004, 25 (5): 515-524. 10.1016/j.cct.2004.04.001.CrossRefPubMed
9.
go back to reference Trowman R, Dumville JC, Torgerson DJ, Cranny G: The impact of trial baseline imbalances should be considered in systematic reviews: a methodological case study. J Clin Epidemiol. 2007, 60 (12): 1229-1233. 10.1016/j.jclinepi.2007.03.014.CrossRefPubMed Trowman R, Dumville JC, Torgerson DJ, Cranny G: The impact of trial baseline imbalances should be considered in systematic reviews: a methodological case study. J Clin Epidemiol. 2007, 60 (12): 1229-1233. 10.1016/j.jclinepi.2007.03.014.CrossRefPubMed
10.
go back to reference Hewitt CE, Kumaravel B, Dumville JC, Torgerson DJ: Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010, 63 (11): 1264-1270. 10.1016/j.jclinepi.2010.01.010.CrossRefPubMed Hewitt CE, Kumaravel B, Dumville JC, Torgerson DJ: Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010, 63 (11): 1264-1270. 10.1016/j.jclinepi.2010.01.010.CrossRefPubMed
11.
go back to reference Nuesch E, Trelle S, Reichenbach S, Rutjes AWS, Burgi E, Scherer M, Altman DG, Juni P: The effects of excluding patients from the analysis in randomized controlled trials: meta-epidemiological study. BMJ. 2009, 339 (7722): 679-683. Nuesch E, Trelle S, Reichenbach S, Rutjes AWS, Burgi E, Scherer M, Altman DG, Juni P: The effects of excluding patients from the analysis in randomized controlled trials: meta-epidemiological study. BMJ. 2009, 339 (7722): 679-683.
12.
go back to reference Bekelman JE, Li Y, Gross CP: Scope and impact of financial conflicts of interest in biomedical research: a systematic review. J Am Med Assoc. 2003, 289 (4): 454-465. 10.1001/jama.289.4.454.CrossRef Bekelman JE, Li Y, Gross CP: Scope and impact of financial conflicts of interest in biomedical research: a systematic review. J Am Med Assoc. 2003, 289 (4): 454-465. 10.1001/jama.289.4.454.CrossRef
13.
go back to reference Lexchin J, Bero LA, Djulbegovic B, Clark O: Pharmaceutical industry sponsorship and research outcome and quality: systematic review. Br Med J. 2003, 326 (7400): 1167-1170. 10.1136/bmj.326.7400.1167.CrossRef Lexchin J, Bero LA, Djulbegovic B, Clark O: Pharmaceutical industry sponsorship and research outcome and quality: systematic review. Br Med J. 2003, 326 (7400): 1167-1170. 10.1136/bmj.326.7400.1167.CrossRef
14.
go back to reference Kjaergard LL, Als-Nielsen B: Association between competing interests and authors’ conclusions: epidemiological study of randomized clinical trials published in the BMJ. Br Med J. 2002, 325 (7358): 249-252. 10.1136/bmj.325.7358.249.CrossRef Kjaergard LL, Als-Nielsen B: Association between competing interests and authors’ conclusions: epidemiological study of randomized clinical trials published in the BMJ. Br Med J. 2002, 325 (7358): 249-252. 10.1136/bmj.325.7358.249.CrossRef
15.
go back to reference Armijo-Olivo S, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ: Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther. 2008, 88 (2): 156-175. 10.2522/ptj.20070147.CrossRef Armijo-Olivo S, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ: Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther. 2008, 88 (2): 156-175. 10.2522/ptj.20070147.CrossRef
16.
go back to reference Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S: Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995, 16 (1): 62-73. 10.1016/0197-2456(94)00031-W.CrossRefPubMed Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S: Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995, 16 (1): 62-73. 10.1016/0197-2456(94)00031-W.CrossRefPubMed
17.
go back to reference Colle F, Rannou F, Revel M, Fermanian J, Poiraudeau S: Impact of quality scales on levels of evidence inferred from a systematic review of exercise therapy and low back pain. Arch Phys Med Rehabil. 2002, 83 (12): 1745-1752. 10.1053/apmr.2002.35657.CrossRefPubMed Colle F, Rannou F, Revel M, Fermanian J, Poiraudeau S: Impact of quality scales on levels of evidence inferred from a systematic review of exercise therapy and low back pain. Arch Phys Med Rehabil. 2002, 83 (12): 1745-1752. 10.1053/apmr.2002.35657.CrossRefPubMed
18.
go back to reference Herbison P, Hay-Smith J, Gillespie WJ: Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol. 2006, 59 (12): 1249.e1-1249.e11. 10.1016/j.jclinepi.2006.03.008.CrossRef Herbison P, Hay-Smith J, Gillespie WJ: Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol. 2006, 59 (12): 1249.e1-1249.e11. 10.1016/j.jclinepi.2006.03.008.CrossRef
19.
go back to reference Armijo-Olivo S, Stiles C, Hagen N, Biondo P, Cummings G: Assessment of study quality for systematic reviews: a comparison of the Cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: methodological research. J Eval Clin Pract. 2010, In press Armijo-Olivo S, Stiles C, Hagen N, Biondo P, Cummings G: Assessment of study quality for systematic reviews: a comparison of the Cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: methodological research. J Eval Clin Pract. 2010, In press
20.
go back to reference Higgins J, Altman D: Chapter 8: assessing risk of bias in included studies. Cochrane handbook for systematic reviews of interventions. Version 5.0 Edition 1 (2009). Edited by: Higgins J, Green S. 2008, Chichester, UK: WileyCrossRef Higgins J, Altman D: Chapter 8: assessing risk of bias in included studies. Cochrane handbook for systematic reviews of interventions. Version 5.0 Edition 1 (2009). Edited by: Higgins J, Green S. 2008, Chichester, UK: WileyCrossRef
21.
go back to reference Kunz R, Autti-Ramo I, Anttila H, Malmivaara A, Makela M: A systematic review finds that methodological quality is better than its reputation but can be improved in physiotherapy trials in childhood cerebral palsy. J Clin Epidemiol. 2006, 59 (12): 1239-1248.CrossRefPubMed Kunz R, Autti-Ramo I, Anttila H, Malmivaara A, Makela M: A systematic review finds that methodological quality is better than its reputation but can be improved in physiotherapy trials in childhood cerebral palsy. J Clin Epidemiol. 2006, 59 (12): 1239-1248.CrossRefPubMed
24.
go back to reference Nuesch E, Reichenbach S, Trelle S, Rutjes AWS, Liewald K, Sterchi R, Altman DG, Juni P: The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. Arthritis Care Res. 2009, 61 (12): 1633-1641. 10.1002/art.24894.CrossRef Nuesch E, Reichenbach S, Trelle S, Rutjes AWS, Liewald K, Sterchi R, Altman DG, Juni P: The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. Arthritis Care Res. 2009, 61 (12): 1633-1641. 10.1002/art.24894.CrossRef
25.
go back to reference Van-Tulder MW, Suttorp M, Morton S, Bouter LM, Shekelle P: Empirical evidence of an association between internal validity and effect size in randomized controlled trials of low-back pain. Spine. 2009, 34 (16): 1685-1692. 10.1097/BRS.0b013e3181ab6a78.CrossRefPubMed Van-Tulder MW, Suttorp M, Morton S, Bouter LM, Shekelle P: Empirical evidence of an association between internal validity and effect size in randomized controlled trials of low-back pain. Spine. 2009, 34 (16): 1685-1692. 10.1097/BRS.0b013e3181ab6a78.CrossRefPubMed
26.
go back to reference Sterne JAC, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M: Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological’ research. Stat Med. 2002, 21 (11): 1513-1524. 10.1002/sim.1184.CrossRefPubMed Sterne JAC, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M: Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological’ research. Stat Med. 2002, 21 (11): 1513-1524. 10.1002/sim.1184.CrossRefPubMed
27.
go back to reference Cohen J: Chapter 1: the concepts of power analysis. Statistical power analysis for the behavioral sciences. Second edition. Edited by: Cohen J. 1988, Hillsdale, NJ: Academic Press Inc, 1-17. Cohen J: Chapter 1: the concepts of power analysis. Statistical power analysis for the behavioral sciences. Second edition. Edited by: Cohen J. 1988, Hillsdale, NJ: Academic Press Inc, 1-17.
28.
go back to reference DerSimonian R, Laird N: Meta-analysis in clinical trials. Control Clin Trials. 1986, 7 (3): 177-188. 10.1016/0197-2456(86)90046-2.CrossRefPubMed DerSimonian R, Laird N: Meta-analysis in clinical trials. Control Clin Trials. 1986, 7 (3): 177-188. 10.1016/0197-2456(86)90046-2.CrossRefPubMed
29.
go back to reference Stevens J: Chapter 11. exploratory and confirmatory factor analysis. Applied multivariate statistics for the social sciences. Edited by: Stevens J. 2002, Mahwah, NJ: L Erlbaum, 385-470. Stevens J: Chapter 11. exploratory and confirmatory factor analysis. Applied multivariate statistics for the social sciences. Edited by: Stevens J. 2002, Mahwah, NJ: L Erlbaum, 385-470.
30.
go back to reference Field AP: Chapter 15: exploratory factor analysis. Discovering statistics using SPSS: (and sex and drugs and rock 'n’ roll). Edited by: Field AP. 2009, Los Angeles (that is, Thousand Oaks, CA); London: SAGE Publications, 619-680. Field AP: Chapter 15: exploratory factor analysis. Discovering statistics using SPSS: (and sex and drugs and rock 'n’ roll). Edited by: Field AP. 2009, Los Angeles (that is, Thousand Oaks, CA); London: SAGE Publications, 619-680.
31.
go back to reference Linstone HA, Turoff M: The Delphi method: techniques and applications. 1975, London: Reading, Mass: Addison-Wesley Pub. Co, Pearson PLC, Advanced Book Program Linstone HA, Turoff M: The Delphi method: techniques and applications. 1975, London: Reading, Mass: Addison-Wesley Pub. Co, Pearson PLC, Advanced Book Program
32.
go back to reference Loo R: The Delphi method: a powerful tool for strategic management. Policing. 2002, 25 (4): 762-769.CrossRef Loo R: The Delphi method: a powerful tool for strategic management. Policing. 2002, 25 (4): 762-769.CrossRef
Metadata
Title
How should we evaluate the risk of bias of physical therapy trials?: a psychometric and meta-epidemiological approach towards developing guidelines for the design, conduct, and reporting of RCTs in Physical Therapy (PT) area: a study protocol
Authors
Susan Armijo-Olivo
Jorge Fuentes
Todd Rogers
Lisa Hartling
Humam Saltaji
Greta G Cummings
Publication date
01-12-2013
Publisher
BioMed Central
Published in
Systematic Reviews / Issue 1/2013
Electronic ISSN: 2046-4053
DOI
https://doi.org/10.1186/2046-4053-2-88

Other articles of this Issue 1/2013

Systematic Reviews 1/2013 Go to the issue