Top

Published in:

Open Access 01-12-2021 | Research

Appraising clinical applicability of studies: mapping and synthesis of current frameworks, and proposal of the FrACAS framework and VICORT checklist

Authors: Quoc Dinh Nguyen, Erica M. Moodie, Philippe Desmarais, Robert Goulden, Marie-France Forget, Eric Peters, Sahar Saeed, Mark R. Keezer, Christina Wolfson

Published in: BMC Medical Research Methodology | Issue 1/2021

Abstract

Background

Not all research findings are translated to clinical practice. Reasons for lack of applicability are varied, and multiple frameworks and criteria exist to appraise the general applicability of epidemiological and clinical research. In this two-part study, we identify, map, and synthesize frameworks and criteria; we develop a framework to assist clinicians to appraise applicability specifically from a clinical perspective.

Methods

We conducted a literature search in PubMed and Embase to identify frameworks appraising applicability of study results. Conceptual thematic analysis was used to synthesize frameworks and criteria. We carried out a framework development process integrating contemporary debates in epidemiology, findings from the literature search and synthesis, iterative pilot-testing, and brainstorming and consensus discussions to propose a concise framework to appraise clinical applicability.

Results

Of the 4622 references retrieved, we identified 26 unique frameworks featuring 21 criteria. Frameworks and criteria varied by scope and level of aggregation of the evidence appraised, target user, and specific area of applicability (internal validity, clinical applicability, external validity, and system applicability). Our proposed Framework Appraising the Clinical Applicability of Studies (FrACAS) classifies studies in three domains (research, practice informing, and practice changing) by examining six criteria sequentially: Validity, Indication-informativeness, Clinical relevance, Originality, Risk-benefit comprehensiveness, and Transposability (VICORT checklist).

Conclusions

Existing frameworks to applicability vary by scope, target user, and area of applicability. We introduce FrACAS to specifically assess applicability from a clinical perspective. Our framework can be used as a tool for the design, appraisal, and interpretation of epidemiological and clinical studies.

Available only for authorised users

Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104:510–20.PubMedPubMedCentralCrossRef

Maclure M, Schneeweiss S. Causation of bias: the episcope. Epidemiology. 2001;12:114–22.PubMedCrossRef

Greenland S. Validity and bias in epidemiological research. Oxford: Oxford University Press; 2011. https://doi.org/10.1093/med/9780199218707.003.0037.CrossRef

Campbell DT. Factors relevant to the validity of experiments in social settings. Psychol Bull. 1957;54:297–312.PubMedCrossRef

Jüni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. Br Med J. 2001;323:42–6.CrossRef

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. https://doi.org/10.1136/bmj.i4919.CrossRefPubMedPubMedCentral

Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898. https://doi.org/10.1136/bmj.l4898.CrossRefPubMed

Greenland S, Morgenstern H. Confounding in health research. Annu Rev Public Health. 2001;22:189–212.PubMedCrossRef

Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology. 2004;15:615–25.PubMedCrossRef

10.

Galea S. An argument for a consequentialist epidemiology. Am J Epidemiol. 2013;178:1185–91.PubMedCrossRef

11.

Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924 LP–926. https://doi.org/10.1136/bmj.39489.470347.AD.CrossRef

12.

Lund H, Juhl CB, Nørgaard B, Draborg E, Henriksen M, Andreasen J, et al. Evidence-based research series-paper 3: using an evidence-based research approach to place your results into context after the study is performed to ensure usefulness of the conclusion. J Clin Epidemiol. 2021;129:167–71. https://doi.org/10.1016/j.jclinepi.2020.07.021.CrossRefPubMed

13.

Methodology Committee of the Patient-Centered Outcomes Research Institute. Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI perspective. JAMA. 2012;307:1636–40.CrossRef

14.

Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 8. Rating the quality of evidence--indirectness. J Clin Epidemiol. 2011;64:1303–10.PubMedCrossRef

15.

Burchett H, Umoquit M, Dobrow M. How do we know when research from one setting can be useful in another? A review of external validity, applicability and transferability frameworks. J Heal Serv Res Policy. 2011;16:238–44.CrossRef

16.

Burchett HED, Blanchard L, Kneale D, Thomas J. Assessing the applicability of public health intervention evaluations from one setting to another: a methodological study of the usability and usefulness of assessment tools and frameworks. Heal Res Policy Syst. 2018;16:15–7.

17.

Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Heal Prof. 2006;29:126–53.CrossRef

18.

Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: A systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016.PubMedCrossRef

19.

Eccles MP, Mittman BS. Welcome to implementation science. Implement Sci. 2006;1:1–3.PubMedCentralCrossRef

20.

Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non-specialist. BMC Psychol. 2015;3:1–12. https://doi.org/10.1186/S40359-015-0089-9.CrossRef

21.

Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. Oxford: Oxford university press; 2015.

22.

Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–7.PubMedPubMedCentralCrossRef

23.

Hernán MA, Taubman SL. Does obesity shorten life? The importance of well-defined interventions to answer causal questions. Int J Obes. 2008;32:S8–14.CrossRef

24.

Daniel RM, De Stavola BL, Vansteelandt S. Commentary: the formal approach to quantitative causal inference in epidemiology: misguided or misrepresented? Int J Epidemiol. 2016;45:1817–29.PubMed

25.

Cole SR, Stuart EA. Generalizing evidence from randomized clinical trials to target populations: the ACTG 320 trial. Am J Epidemiol. 2010;172:107–15.PubMedPubMedCentralCrossRef

26.

Hernán MA, Robins JM. Causal Inference : what if. Boca Raton: Chapman & Hall/CRC; 2020.

27.

Fleming TR. Surrogate endpoints and FDA’s accelerated approval process. Health Aff (Millwood). 2005;24:67–78. https://doi.org/10.1377/hlthaff.24.1.67.CrossRef

28.

Creswell JW, Creswell JD. Research design: qualitative, quantitative, and mixed methods approaches. Thousand Oaks: Sage Publications; 2017.

29.

Cluzeau F, Burgers J, Brouwers M, Grol R, Mäkelä M, Littlejohns P, et al. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Heal Care. 2003;12:18–23.CrossRef

30.

Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. Cmaj. 2010;182:839–42.CrossRef

31.

Brouwers MC, Spithoff K, Kerkvliet K, Alonso-Coello P, Burgers J, Cluzeau F, et al. Development and validation of a tool to assess the quality of clinical practice guideline recommendations. JAMA Netw Open. 2020;3:e205535.PubMedPubMedCentralCrossRef

32.

Cambon L, Minary L, Ridde V, Alla F. A tool to analyze the transferability of health promotion interventions. BMC Public Health. 2013;13:1184. https://doi.org/10.1186/1471-2458-13-1184.CrossRefPubMedPubMedCentral

33.

Khorsan R, Crawford C. External validity and model validity: a conceptual approach for systematic review methodology. Evid Based Complement Alternat Med. 2014;2014:1–12.CrossRef

34.

Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: A systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ. 2016;353:i2089.PubMedCrossRef

35.

Parmelli E, Amato L, Oxman AD, Alonso-Coello P, Brunetti M, Moberg J, et al. GRADE evidence to decision (EtD) framework for coverage decisions. Int J Technol Assess Health Care. 2017;33:176–82.PubMedCrossRef

36.

Schünemann HJ, Wiercioch W, Brozek J, Etxeandia-Ikobaltzeta I, Mustafa RA, Manja V, et al. GRADE evidence to decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol. 2017;81:101–10.PubMedCrossRef

37.

Moberg J, Oxman AD, Rosenbaum S, Schünemann HJ, Guyatt G, Flottorp S, et al. The GRADE evidence to decision (EtD) framework for health system and public health decisions. Heal Res Policy Syst. 2018;16:1–15.

38.

Piggott T, Brozek J, Nowak A, Dietl H, Dietl B, Saz-Parkinson Z, et al. Using GRADE evidence to decision frameworks to choose from multiple interventions. J Clin Epidemiol. 2021;130:117–24. https://doi.org/10.1016/j.jclinepi.2020.10.016.CrossRefPubMed

39.

Khalifa M, Magrabi F, Gallego B. Developing a framework for evidence-based grading and assessment of predictive tools for clinical decision support. BMC Med Inform Decis Mak. 2019;19:207. https://doi.org/10.1186/s12911-019-0940-7.CrossRefPubMedPubMedCentral

40.

Milat A, Lee K, Conte K, Grunseit A, Wolfenden L, Van Nassau F, et al. Intervention scalability assessment tool: a decision support tool for health policy makers and implementers. Heal Res Policy Syst. 2020;18:1–17.CrossRef

41.

Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. C Can Med Assoc J. 2009;180(10):E47-57.

42.

Koppenaal T, Linmans J, Knottnerus JA, Spigt M. Pragmatic vs. explanatory: an adaptation of the PRECIS tool helps to judge the applicability of systematic reviews for daily practice. J Clin Epidemiol. 2011;64:1095–101.PubMedCrossRef

43.

Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med. 2019;170:W1–33.PubMedCrossRef

44.

Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res. 2006;21:688–94.PubMedCrossRef

45.

Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 13: Preparing and using policy briefs to support evidence-informed policymaking. Heal Res Policy Syst. 2009;7(Suppl 1):S13.CrossRef

46.

Stratil JM, Baltussen R, Scheel I, Nacken A, Rehfuess EA. Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making. Cost Eff Resour Alloc. 2020;18:1–15. https://doi.org/10.1186/s12962-020-0203-6.CrossRef

47.

Almeida ND, Mines L, Nicolau I, Sinclair A, Forero DF, Brophy JM, et al. A framework for aiding the translation of scientific evidence into policy: the experience of a hospital-based technology assessment unit. Int J Technol Assess Health Care. 2019;35:204–11.PubMedCrossRef

48.

Atkins D, Chang SM, Gartlehner G, Buckley DI, Whitlock EP, Berliner E, et al. Assessing applicability when comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol. 2011;64:1198–207. https://doi.org/10.1016/j.jclinepi.2010.11.021.CrossRefPubMed

49.

Berger ML, Martin BC, Husereau D, Worley K, Allen JD, Yang W, et al. A questionnaire to assess the relevance and credibility of observational studies to inform health care decision making: an ISPOR-AMCP-NPC good practice task force report. Value Health. 2014;17:143–56. https://doi.org/10.1016/j.jval.2013.12.011.CrossRefPubMedPubMedCentral

50.

Bonell C, Oakley A, Hargreaves J, Strange V, Rees R. Assessment of generalisability in trials of health interventions: suggested framework and systematic review. BMJ. 2006;333:346 LP–349. https://doi.org/10.1136/bmj.333.7563.346.CrossRef

51.

Bornhöft G, Maxion-Bergemann S, Wolf U, Kienle GS, Michalsen A, Vollmar HC, et al. Checklist for the qualitative evaluation of clinical studies with particular focus on external validity and model validity. BMC Med Res Methodol. 2006;6:1–13.CrossRef

52.

Burford B, Lewin S, Welch V, Rehfuess E, Waters E. Assessing the applicability of findings in systematic reviews of complex interventions can enhance the utility of reviews for decision making. J Clin Epidemiol. 2013;66:1251–61. https://doi.org/10.1016/j.jclinepi.2013.06.017.CrossRefPubMed

53.

Gruen RL, Morris PS, McDonald EL, Bailie RS. Making systematic reviews more useful for policy-makers. Bull World Health Organ. 2005;83:480.PubMedPubMedCentral

54.

Linan Z, Qiusha Y, Chuan Z, Chao H, Hailong L, Chunsong Y, et al. An instrument for evaluating the clinical applicability of guidelines. J Evid Based Med. 2020;14(1):75–81.PubMedCentralCrossRef

55.

Polus S, Pfadenhauer L, Brereton L, Leppert W, Wahlster P, Gerhardus A, et al. A consultation guide for assessing the applicability of health technologies: a case study. Int J Technol Assess Health Care. 2017;33:577–85.PubMedCrossRef

56.

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:1–7.

57.

Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting, and evaluation in health care. Prev Med (Baltim). 2010;51:421–4. https://doi.org/10.1016/j.ypmed.2010.08.005.CrossRef

58.

Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses 2000.

59.

Lewis D. Counterfactuals. Hoboken: Wiley; 2013.

60.

Höfler M. Causal inference based on counterfactuals. BMC Med Res Methodol. 2005;5:1–12.CrossRef

61.

Cole SR, Frangakis CE. The consistency statement in causal inference: a definition or an assumption? Epidemiology. 2009;20:3–5.PubMedCrossRef

62.

Fleming TR, Powers JH. Biomarkers and surrogate endpoints in clinical trials. Stat Med. 2012;31:2973–84. https://doi.org/10.1002/sim.5403.CrossRefPubMedPubMedCentral

63.

Brodersen J, Schwartz LM, Heneghan C, O’Sullivan JW, Aronson JK, Woloshin S. Overdiagnosis: what it is and what it isn’t. BMJ Evid Based Med. 2018;23:1–3. https://doi.org/10.1136/ebmed-2017-110886.CrossRefPubMed

64.

Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. Am Stat. 2016;70:129–33. https://doi.org/10.1080/00031305.2016.1154108.CrossRef

65.

Ioannidis JPA. The Proposal to Lower P Value Thresholds to .005. JAMA. 2018;319:1429–30. https://doi.org/10.1001/jama.2018.1536.CrossRefPubMed

66.

Poole C. On the origin of risk relativism. Epidemiology. 2010;21:3–9.PubMedCrossRef

67.

Noordzij M, Van Diepen M, Caskey FC, Jager KJ. Relative risk versus absolute risk: One cannot be interpreted without the other. Nephrol Dial Transplant. 2017;32:ii13–8.PubMedCrossRef

68.

de Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in medicine: a practical guide. Cambridge: Cambridge University Press; 2011. https://doi.org/10.1017/CBO9780511996214.CrossRef

69.

Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 2015;49:560–75. https://doi.org/10.1111/medu.12678.CrossRefPubMed

70.

Kelley TL. Interpretation of educational measurements; 1927.

71.

Hernán MA. The C-word: scientific euphemisms do not improve causal inference from observational data. Am J Public Health. 2018;108:616–9.PubMedPubMedCentralCrossRef

72.

Vandenbroucke JP, Broadbent A, Pearce N. Causality and causal inference in epidemiology: the need for a pluralistic approach. Int J Epidemiol. 2016;45:1776–86.PubMedPubMedCentralCrossRef

73.

Krieger N, Smith GD. The tale wagged by the DAG: broadening the scope of causal inference and explanation for epidemiology. Int J Epidemiol. 2016;45:1787–808.PubMed

74.

VanderWeele TJ, Hernán MA, Tchetgen Tchetgen EJ, Robins JM. Re: causality and causal inference in epidemiology: the need for a pluralistic approach. Int J Epidemiol. 2016;45:2199–200.PubMed

75.

Glymour C, Glymour MR. Commentary: race and sex are causes. Epidemiology. 2014;25:488–90. https://doi.org/10.1097/EDE.0000000000000122.CrossRefPubMed

76.

Lund H, Juhl CB, Nørgaard B, Draborg E, Henriksen M, Andreasen J, et al. Evidence-Based Research Series-Paper 2 : Using an Evidence-Based Research approach before a new study is conducted to ensure value. J Clin Epidemiol. 2021;129:158–66. https://doi.org/10.1016/j.jclinepi.2020.07.019.CrossRefPubMed

77.

Hunink MGM, Weinstein MC, Wittenberg E, Drummond MF, Pliskin JS, Wong JB, et al. Decision making in health and medicine: integrating evidence and values. 2nd ed. Cambridge: Cambridge University Press; 2014. https://doi.org/10.1017/CBO9781139506779.CrossRef

78.

Singh D. Merck withdraws arthritis drug worldwide. BMJ. 2004;329:816. https://doi.org/10.1136/bmj.329.7470.816-a.CrossRefPubMedPubMedCentral

79.

Hernán MA, Vanderweele TJ. Compound treatments and transportability of causal inference. Epidemiology. 2011;22:368–77.PubMedPubMedCentralCrossRef

80.

Bareinboim E, Pearl J. Causal inference and the data-fusion problem. Proc Natl Acad Sci. 2016;113:7345–52. https://doi.org/10.1073/pnas.1510507113.CrossRefPubMedPubMedCentral

81.

Cambon L, Minary L, Ridde V, Alla F. A tool to facilitate transferability of health promotion interventions: ASTAIRE. Sante Publique (Paris). 2014;26:783–6.CrossRef

82.

Viswanathan M, Ansari MT, Berkman ND, Chang S, Hartling L, McPheeters M, et al. Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville; 2012.

83.

Baker A, Potter J, Young K, Madan I. The applicability of grading systems for guidelines. J Eval Clin Pract. 2011;17:758–62.PubMedCrossRef

84.

Siering U, Eikermann M, Hausner E, Hoffmann-Eßer W, Neugebauer EA. Appraisal tools for clinical practice guidelines: a systematic review. PLOS ONE. 2013;8(12):e82915.

85.

Graham ID, Calder LA, Hébert PC, Carter AO, Tetroe JM. A comparison of clinical practice guideline appraisal instruments. Int J Technol Assess Health Care. 2000;16:1024–38.PubMedCrossRef

Title: Appraising clinical applicability of studies: mapping and synthesis of current frameworks, and proposal of the FrACAS framework and VICORT checklist
Authors: Quoc Dinh Nguyen
Erica M. Moodie
Philippe Desmarais
Robert Goulden
Marie-France Forget
Eric Peters
Sahar Saeed
Mark R. Keezer
Christina Wolfson
Publication date: 01-12-2021
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2021
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/s12874-021-01445-0

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Appraising clinical applicability of studies: mapping and synthesis of current frameworks, and proposal of the FrACAS framework and VICORT checklist

Abstract

Background

Methods

Results

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2021

New adaptive lasso approaches for variable selection in automated pharmacovigilance signal detection

Using electronic health records to inform trial feasibility in a rare autoimmune blistering skin disease in England

Evaluating the relationship between citation set size, team size and screening methods used in systematic reviews: a cross-sectional study

A new hybrid record linkage process to make epidemiological databases interoperable: application to the GEMO and GENEPSO studies involving BRCA1 and BRCA2 mutation carriers

National surveillance of stroke quality of care and outcomes by applying post-stratification survey weights on the Get With The Guidelines-Stroke patient registry

Ethnographic research as an evolving method for supporting healthcare improvement skills: a scoping review