Skip to main content
Top
Published in: International Journal of Clinical Pharmacy 3/2019

01-06-2019 | Research Article

Usability and sensitivity of the risk of bias assessment tool for randomized controlled trials of pharmacist interventions

Authors: Fernanda S. Tonin, Livia A. Lopes, Inajara Rotta, Aline F. Bonetti, Roberto Pontarolo, Cassyano J. Correr, Fernando Fernandez-Llimos

Published in: International Journal of Clinical Pharmacy | Issue 3/2019

Login to get access

Abstract

Background The Cochrane collaboration risk of bias assessment (RoB) tool is used in several fields to evaluate the methodological quality of studies. Its strengths and challenges are discussed. Objective To assess the sensitivity of the RoB tool in studies of pharmacist interventions. Setting DEPICT database was used to pool randomized controlled trials (RCTs) of complex interventions. Method A Guide for RoB Judgment in Pharmacy Services was created to help in the interpretation and judgment of bias criteria. The evaluation of bias (low, unclear, high risk) was performed by RCT. Sensitivity analyses were performed to assess the influence of different interpretations of eight elements of judgment in the RoB tool. Paired analysis and estimations of the effect size (95% confidence interval) of the criteria modifications compared to the original analyses were calculated. Main outcome measure Changes in the interpretations of judgment in the RoB tool. Results Overall, 8.3, 45.4, and 46.3% of the studies were determined to have low, unclear, and high risk of bias, respectively. High risk of bias was caused by attrition and detection domains. The number of studies classified with high risk of bias significantly increased for five of the eight interpretations, while unclear risk of bias increased for three interpretations (with a negligible effect size in all of them). Lack of blinding, loss of participants, and the use of subjective and self-reported outcomes were the main elements resulting in high risk of bias. Conclusion The RoB tool is useful for evaluating RCTs of pharmacist interventions if adapted criteria for judgment are used. Ignoring these adjustments produces a floor-effect with studies classified with high risk of bias.
Appendix
Available only for authorised users
Literature
1.
go back to reference Armijo-Olivo S, Fuentes J, Ospina M, Saltaji H, Hartling L. Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: a descriptive analysis. BMC Med Res Methodol. 2013;13:116.CrossRefPubMedPubMedCentral Armijo-Olivo S, Fuentes J, Ospina M, Saltaji H, Hartling L. Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: a descriptive analysis. BMC Med Res Methodol. 2013;13:116.CrossRefPubMedPubMedCentral
2.
go back to reference Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet. 2017;390:415–23.CrossRefPubMed Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet. 2017;390:415–23.CrossRefPubMed
3.
go back to reference Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs Seida J, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012.CrossRefPubMedPubMedCentral Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs Seida J, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012.CrossRefPubMedPubMedCentral
4.
go back to reference Viswanathan M, Ansari MT, Berkman ND, Chang S, Hartling L, McPheeters M, et al. Assessing the risk of bias of individual studies in systematic reviews of health care interventions. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ: Rockville, 2008. Viswanathan M, Ansari MT, Berkman ND, Chang S, Hartling L, McPheeters M, et al. Assessing the risk of bias of individual studies in systematic reviews of health care interventions. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ: Rockville, 2008.
6.
go back to reference Sylvester RJ, Canfield SE, Lam TB, Marconi L, MacLennan S, Yuan Y, et al. Conflict of evidence: resolving discrepancies when findings from randomized controlled trials and meta-analyses disagree. Eur Urol. 2017;71:811–9.CrossRefPubMed Sylvester RJ, Canfield SE, Lam TB, Marconi L, MacLennan S, Yuan Y, et al. Conflict of evidence: resolving discrepancies when findings from randomized controlled trials and meta-analyses disagree. Eur Urol. 2017;71:811–9.CrossRefPubMed
7.
go back to reference Zeng X, Zhang Y, Kwong JS, Zhang C, Li S, Sun F, et al. The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: a systematic review. J Evid Based Med. 2015;8:2–10.CrossRefPubMed Zeng X, Zhang Y, Kwong JS, Zhang C, Li S, Sun F, et al. The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: a systematic review. J Evid Based Med. 2015;8:2–10.CrossRefPubMed
8.
go back to reference Jordan VM, Lensen SF, Farquhar CM. There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review. J Clin Epidemiol. 2017;81:72–6.CrossRefPubMed Jordan VM, Lensen SF, Farquhar CM. There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review. J Clin Epidemiol. 2017;81:72–6.CrossRefPubMed
9.
go back to reference Mansournia MA, Higgins JP, Sterne JA, Hernan MA. Biases in randomized trials: a conversation between trialists and epidemiologists. Epidemiology. 2017;28:54–9.CrossRefPubMedPubMedCentral Mansournia MA, Higgins JP, Sterne JA, Hernan MA. Biases in randomized trials: a conversation between trialists and epidemiologists. Epidemiology. 2017;28:54–9.CrossRefPubMedPubMedCentral
10.
go back to reference Yamato TP, Maher C, Koes B, Moseley A. The PEDro scale had acceptably high convergent validity, construct validity and inter-rater reliability in evaluating methodological quality of pharmaceutical trials. J Clin Epidemiol. 2017;86:176–81.CrossRefPubMed Yamato TP, Maher C, Koes B, Moseley A. The PEDro scale had acceptably high convergent validity, construct validity and inter-rater reliability in evaluating methodological quality of pharmaceutical trials. J Clin Epidemiol. 2017;86:176–81.CrossRefPubMed
11.
go back to reference Hempel S, Miles JN, Booth MJ, Wang Z, Morton SC, Shekelle PG. Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis. Syst Rev. 2013;2:107.CrossRefPubMedPubMedCentral Hempel S, Miles JN, Booth MJ, Wang Z, Morton SC, Shekelle PG. Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis. Syst Rev. 2013;2:107.CrossRefPubMedPubMedCentral
12.
go back to reference Saltaji H, Ospina MB, Armijo-Olivo S, Agarwal S, Cummings GG, Amin M, et al. Evaluation of risk of bias assessment of trials in systematic reviews of oral health interventions, 1991–2014: a methodology study. J Am Dent Assoc. 2016;147:720–8.CrossRefPubMed Saltaji H, Ospina MB, Armijo-Olivo S, Agarwal S, Cummings GG, Amin M, et al. Evaluation of risk of bias assessment of trials in systematic reviews of oral health interventions, 1991–2014: a methodology study. J Am Dent Assoc. 2016;147:720–8.CrossRefPubMed
13.
go back to reference Vale CL, Tierney JF, Burdett S. Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews. BMJ. 2013;346:f1798.CrossRefPubMed Vale CL, Tierney JF, Burdett S. Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews. BMJ. 2013;346:f1798.CrossRefPubMed
14.
go back to reference Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG. Assessment of study quality for systematic reviews: a comparison of the Cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: methodological research. J Eval Clin Pract. 2012;18:12–8.CrossRefPubMed Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG. Assessment of study quality for systematic reviews: a comparison of the Cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: methodological research. J Eval Clin Pract. 2012;18:12–8.CrossRefPubMed
15.
go back to reference Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther. 2008;88:156–75.CrossRefPubMed Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther. 2008;88:156–75.CrossRefPubMed
17.
go back to reference da Costa BR, Beckett B, Diaz A, Resta NM, Johnston BC, Egger M, et al. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study. Syst Rev. 2017;6:44.CrossRefPubMedPubMedCentral da Costa BR, Beckett B, Diaz A, Resta NM, Johnston BC, Egger M, et al. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study. Syst Rev. 2017;6:44.CrossRefPubMedPubMedCentral
18.
go back to reference Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013;66:973–81.CrossRefPubMed Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013;66:973–81.CrossRefPubMed
19.
go back to reference Manchikanti L, Hirsch JA, Cohen SP, Heavner JE, Falco FJ, Diwan S, et al. Assessment of methodologic quality of randomized trials of interventional techniques: development of an interventional pain management specific instrument. Pain Physician. 2014;17:E263–90.PubMed Manchikanti L, Hirsch JA, Cohen SP, Heavner JE, Falco FJ, Diwan S, et al. Assessment of methodologic quality of randomized trials of interventional techniques: development of an interventional pain management specific instrument. Pain Physician. 2014;17:E263–90.PubMed
20.
go back to reference Jorgensen L, Paludan-Muller AS, Laursen DR, Savovic J, Boutron I, Sterne JA, et al. Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews. Syst Rev. 2016;5:80.CrossRefPubMedPubMedCentral Jorgensen L, Paludan-Muller AS, Laursen DR, Savovic J, Boutron I, Sterne JA, et al. Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews. Syst Rev. 2016;5:80.CrossRefPubMedPubMedCentral
21.
go back to reference Higgins J, Sterne J, Savović J, Page M, Hróbjartsson A, Boutron I, et al. A revised tool for assessing risk of bias in randomized trials. Cochrane Database Syst Rev. 2016;10:30–1. Higgins J, Sterne J, Savović J, Page M, Hróbjartsson A, Boutron I, et al. A revised tool for assessing risk of bias in randomized trials. Cochrane Database Syst Rev. 2016;10:30–1.
22.
go back to reference Vo TH, Charpiat B, Catoire C, Juste M, Roubille R, Rose FX, et al. Tools for assessing potential significance of pharmacist interventions: a systematic review. Drug Saf. 2016;39:131–46.CrossRefPubMed Vo TH, Charpiat B, Catoire C, Juste M, Roubille R, Rose FX, et al. Tools for assessing potential significance of pharmacist interventions: a systematic review. Drug Saf. 2016;39:131–46.CrossRefPubMed
24.
go back to reference Correr CJ, Melchiors AC, de Souza TT, Rotta I, Salgado TM, Fernandez-Llimos F. A tool to characterize the components of pharmacist interventions in clinical pharmacy services: the DEPICT project. Ann Pharmacother. 2013;47:946–52.CrossRefPubMed Correr CJ, Melchiors AC, de Souza TT, Rotta I, Salgado TM, Fernandez-Llimos F. A tool to characterize the components of pharmacist interventions in clinical pharmacy services: the DEPICT project. Ann Pharmacother. 2013;47:946–52.CrossRefPubMed
25.
go back to reference Rotta I, Salgado TM, Felix DC, Souza TT, Correr CJ, Fernandez-Llimos F. Ensuring consistent reporting of clinical pharmacy services to enhance reproducibility in practice: an improved version of DEPICT. J Eval Clin Pract. 2015;21:584–90.CrossRefPubMed Rotta I, Salgado TM, Felix DC, Souza TT, Correr CJ, Fernandez-Llimos F. Ensuring consistent reporting of clinical pharmacy services to enhance reproducibility in practice: an improved version of DEPICT. J Eval Clin Pract. 2015;21:584–90.CrossRefPubMed
26.
go back to reference Rotta I, Souza TT, Salgado TM, Correr CJ, Fernandez-Llimos F. Characterization of published randomized controlled trials assessing clinical pharmacy services around the world. Res Soc Adm Pharm. 2017;13:201–8.CrossRef Rotta I, Souza TT, Salgado TM, Correr CJ, Fernandez-Llimos F. Characterization of published randomized controlled trials assessing clinical pharmacy services around the world. Res Soc Adm Pharm. 2017;13:201–8.CrossRef
27.
go back to reference da Costa BR, Resta NM, Beckett B, Israel-Stahre N, Diaz A, Johnston BC, et al. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol. Syst Rev. 2014;13(3):144.CrossRef da Costa BR, Resta NM, Beckett B, Israel-Stahre N, Diaz A, Johnston BC, et al. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol. Syst Rev. 2014;13(3):144.CrossRef
28.
go back to reference Moustgaard H, Bello S, Miller FG, Hrobjartsson A. Subjective and objective outcomes in randomized clinical trials: definitions differed in methods publications and were often absent from trial reports. J Clin Epidemiol. 2014;67:1327–34.CrossRefPubMed Moustgaard H, Bello S, Miller FG, Hrobjartsson A. Subjective and objective outcomes in randomized clinical trials: definitions differed in methods publications and were often absent from trial reports. J Clin Epidemiol. 2014;67:1327–34.CrossRefPubMed
30.
go back to reference Cliff N. Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull. 1993;114:494–509.CrossRef Cliff N. Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull. 1993;114:494–509.CrossRef
31.
go back to reference Dreier M, Borutta B, Stahmeyer J, Krauth C, Walter U. Comparison of tools for assessing the methodological quality of primary and secondary studies in health technology assessment reports in Germany. GMS Health Technol Assess. 2010;6:Doc07.PubMedPubMedCentral Dreier M, Borutta B, Stahmeyer J, Krauth C, Walter U. Comparison of tools for assessing the methodological quality of primary and secondary studies in health technology assessment reports in Germany. GMS Health Technol Assess. 2010;6:Doc07.PubMedPubMedCentral
32.
go back to reference Armijo-Olivo S, da Costa BR, Cummings GG, Ha C, Fuentes J, Saltaji H, et al. PEDro or Cochrane to assess the quality of clinical trials? A meta-epidemiological study. PLoS ONE. 2015;10:e0132634.CrossRefPubMedPubMedCentral Armijo-Olivo S, da Costa BR, Cummings GG, Ha C, Fuentes J, Saltaji H, et al. PEDro or Cochrane to assess the quality of clinical trials? A meta-epidemiological study. PLoS ONE. 2015;10:e0132634.CrossRefPubMedPubMedCentral
33.
go back to reference Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282:1054–60.CrossRefPubMed Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282:1054–60.CrossRefPubMed
34.
go back to reference Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ. 2004;328:22–4.CrossRefPubMedPubMedCentral Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ. 2004;328:22–4.CrossRefPubMedPubMedCentral
35.
go back to reference Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;2010(340):c332.CrossRef Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;2010(340):c332.CrossRef
36.
go back to reference Dechartres A, Trinquart L, Faber T, Ravaud P. Empirical evaluation of which trial characteristics are associated with treatment effect estimates. J Clin Epidemiol. 2016;77:24–37.CrossRefPubMed Dechartres A, Trinquart L, Faber T, Ravaud P. Empirical evaluation of which trial characteristics are associated with treatment effect estimates. J Clin Epidemiol. 2016;77:24–37.CrossRefPubMed
37.
go back to reference Glynn LG, Murphy AW, Smith SM, Schroeder K, Fahey T. Self-monitoring and other non-pharmacological interventions to improve the management of hypertension in primary care: a systematic review. Br J Gen Pract. 2010;60:e476–88.CrossRefPubMedPubMedCentral Glynn LG, Murphy AW, Smith SM, Schroeder K, Fahey T. Self-monitoring and other non-pharmacological interventions to improve the management of hypertension in primary care: a systematic review. Br J Gen Pract. 2010;60:e476–88.CrossRefPubMedPubMedCentral
38.
go back to reference Bos JM, van den Bemt PM, de Smet PA, Kramers C. The effect of prescriber education on medication-related patient harm in the hospital: a systematic review. Br J Clin Pharmacol. 2017;83:953–61.CrossRefPubMedPubMedCentral Bos JM, van den Bemt PM, de Smet PA, Kramers C. The effect of prescriber education on medication-related patient harm in the hospital: a systematic review. Br J Clin Pharmacol. 2017;83:953–61.CrossRefPubMedPubMedCentral
39.
go back to reference Clark L, Fairhurst C, Torgerson DJ. Allocation concealment in randomised controlled trials: Are we getting better? BMJ. 2016;355:i5663.CrossRefPubMed Clark L, Fairhurst C, Torgerson DJ. Allocation concealment in randomised controlled trials: Are we getting better? BMJ. 2016;355:i5663.CrossRefPubMed
41.
42.
go back to reference Fielding S, Ogbuagu A, Sivasubramaniam S, MacLennan G, Ramsay CR. Reporting and dealing with missing quality of life data in RCTs: has the picture changed in the last decade? Qual Life Res. 2016;25:2977–83.CrossRefPubMedPubMedCentral Fielding S, Ogbuagu A, Sivasubramaniam S, MacLennan G, Ramsay CR. Reporting and dealing with missing quality of life data in RCTs: has the picture changed in the last decade? Qual Life Res. 2016;25:2977–83.CrossRefPubMedPubMedCentral
43.
go back to reference Wood AM, White IR, Thompson SG. Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clin Trials. 2004;1:368–76.CrossRefPubMed Wood AM, White IR, Thompson SG. Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clin Trials. 2004;1:368–76.CrossRefPubMed
44.
go back to reference Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ. 2015;350:h391.CrossRefPubMed Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ. 2015;350:h391.CrossRefPubMed
45.
go back to reference Hewitt CE, Kumaravel B, Dumville JC, Torgerson DJ, Trial attrition study group. Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010;63:1264–70.CrossRefPubMed Hewitt CE, Kumaravel B, Dumville JC, Torgerson DJ, Trial attrition study group. Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010;63:1264–70.CrossRefPubMed
46.
go back to reference Jarvinen TL, Sihvonen R, Bhandari M, Sprague S, Malmivaara A, Paavola M, et al. Blinded interpretation of study results can feasibly and effectively diminish interpretation bias. J Clin Epidemiol. 2014;67:769–72.CrossRefPubMed Jarvinen TL, Sihvonen R, Bhandari M, Sprague S, Malmivaara A, Paavola M, et al. Blinded interpretation of study results can feasibly and effectively diminish interpretation bias. J Clin Epidemiol. 2014;67:769–72.CrossRefPubMed
47.
go back to reference Armijo-Olivo S, Ospina M, da Costa BR, Egger M, Saltaji H, Fuentes J, et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS ONE. 2014;9:e96920.CrossRefPubMedPubMedCentral Armijo-Olivo S, Ospina M, da Costa BR, Egger M, Saltaji H, Fuentes J, et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS ONE. 2014;9:e96920.CrossRefPubMedPubMedCentral
48.
go back to reference Greenland S, Schlesselman JJ, Criqui MH. The fallacy of employing standardized regression coefficients and correlations as measures of effect. Am J Epidemiol. 1986;123:203–8.CrossRefPubMed Greenland S, Schlesselman JJ, Criqui MH. The fallacy of employing standardized regression coefficients and correlations as measures of effect. Am J Epidemiol. 1986;123:203–8.CrossRefPubMed
Metadata
Title
Usability and sensitivity of the risk of bias assessment tool for randomized controlled trials of pharmacist interventions
Authors
Fernanda S. Tonin
Livia A. Lopes
Inajara Rotta
Aline F. Bonetti
Roberto Pontarolo
Cassyano J. Correr
Fernando Fernandez-Llimos
Publication date
01-06-2019
Publisher
Springer International Publishing
Published in
International Journal of Clinical Pharmacy / Issue 3/2019
Print ISSN: 2210-7703
Electronic ISSN: 2210-7711
DOI
https://doi.org/10.1007/s11096-019-00818-2

Other articles of this Issue 3/2019

International Journal of Clinical Pharmacy 3/2019 Go to the issue