Skip to main content
Top

Open Access 12-03-2025 | Original Research Article

Using Mixed Methods to Evaluate Risk Minimisation Programs in Europe and the USA: An Innovative Blueprint

Authors: Meredith Y. Smith, Rachel Davis, Priya Bahri, Delphine Saragoussi, Viviana Nguyen, Gita A. Toyserkani, Alison Hamilton

Published in: Drug Safety

Login to get access

Abstract

Background

Significant methodological shortcomings have been documented to date in risk minimisation program evaluations for medicinal products, including overreliance on survey methods alone. Recently updated guidances from the Food and Drug Administration (FDA) and the European Medicines Agency (EMA) recommend the use of frameworks and mixed methods designs to improve the rigor of these assessments.

Objective

The purpose of this paper was to exemplify how a mixed methods approach, guided by an implementation science framework, can be used to design the evaluation of a risk minimisation program.

Methods

We selected the Practical, Robust, Implementation and Sustainability Model (PRISM) as the implementation science framework to guide our mixed methods approach. PRISM provides a comprehensive and systematic approach to measuring the key domains relevant to the implementation and outcomes of a risk minimisation program. We mapped the PRISM domains to the evaluation dimensions described in the EMA and FDA guidances. We then specified a mixed methods evaluation design and data collection methods using a fictitious risk minimisation program as a case study for illustrative purposes.

Results

On the basis of our case study, we developed quantitative and qualitative measures, including specific items for surveys and interviews, for both formative and summative evaluations. For both the formative and summative evaluations, measures focussed on assessing (1) contextual factors that could affect program implementation and impact and (2) outcomes including implementability and acceptability as well as degree of program reach, adoption, implementation, effectiveness and maintenance.

Conclusions

Mixed methods, guided by a well-established implementation science framework, can be applied to ensure comprehensive formative and summative evaluations that provide fit-for-purpose information that may inform regulatory decision-making.
Literature
6.
go back to reference Sarpatwari A, Mitra-Majumdar M, Bykov K, Avorn J, Woloshin S, Toyserkani GA, et al. A multi-modal approach to evaluate the impact of risk evaluation and mitigation strategy (REMS) programs. Drug Saf. 2021;44(7):743–51.PubMedCrossRef Sarpatwari A, Mitra-Majumdar M, Bykov K, Avorn J, Woloshin S, Toyserkani GA, et al. A multi-modal approach to evaluate the impact of risk evaluation and mitigation strategy (REMS) programs. Drug Saf. 2021;44(7):743–51.PubMedCrossRef
7.
go back to reference Bahri P. Public pharmacovigilance communication: a process calling for evidence-based, objective-driven strategies. Drug Saf. 2010;33(12):1065–79.PubMedCrossRef Bahri P. Public pharmacovigilance communication: a process calling for evidence-based, objective-driven strategies. Drug Saf. 2010;33(12):1065–79.PubMedCrossRef
8.
go back to reference Morrato EH, Smith MY. Dissemination and implementation science. In: Bahri P, editor. Communicating about risks and safe use of medicines: real life and applied research. Berlin: Springer; 2020. p. 385–413.CrossRef Morrato EH, Smith MY. Dissemination and implementation science. In: Bahri P, editor. Communicating about risks and safe use of medicines: real life and applied research. Berlin: Springer; 2020. p. 385–413.CrossRef
9.
go back to reference Smith MY, Hogan SA, Jack SM, Wilson RT, Oremus M. The case for using mixed methods for designing, implementing, and disseminating evidence-based interventions for public health practice. J Public Health Policy. 2022;43(2):292–303.PubMedCrossRef Smith MY, Hogan SA, Jack SM, Wilson RT, Oremus M. The case for using mixed methods for designing, implementing, and disseminating evidence-based interventions for public health practice. J Public Health Policy. 2022;43(2):292–303.PubMedCrossRef
11.
go back to reference Morrato EH. Social science theory as a framework for designing and evaluating pharmaceutical risk management dissemination and implementation Strategies. In: Sprafka JM, editor. Risk management principles for devices and pharmaceuticals. 3rd ed. Rockville: Regulatory Affairs Professionals Society; 2023. Morrato EH. Social science theory as a framework for designing and evaluating pharmaceutical risk management dissemination and implementation Strategies. In: Sprafka JM, editor. Risk management principles for devices and pharmaceuticals. 3rd ed. Rockville: Regulatory Affairs Professionals Society; 2023.
12.
go back to reference Huynh L, Toyserkani GA, Morrato EH. Pragmatic applications of implementation science frameworks to regulatory science: an assessment of FDA risk evaluation and mitigation strategies (REMS) (2014–2018). BMC Health Serv Res. 2021;21(1):779.PubMedPubMedCentralCrossRef Huynh L, Toyserkani GA, Morrato EH. Pragmatic applications of implementation science frameworks to regulatory science: an assessment of FDA risk evaluation and mitigation strategies (REMS) (2014–2018). BMC Health Serv Res. 2021;21(1):779.PubMedPubMedCentralCrossRef
13.
go back to reference Toyserkani GA, Huynh L, Morrato EH. Adaptation for regulatory application: a content analysis of FDA risk evaluation and mitigation strategies assessment plans (2014–2018) using RE-AIM. Front Public Health. 2020;8:43.PubMedPubMedCentralCrossRef Toyserkani GA, Huynh L, Morrato EH. Adaptation for regulatory application: a content analysis of FDA risk evaluation and mitigation strategies assessment plans (2014–2018) using RE-AIM. Front Public Health. 2020;8:43.PubMedPubMedCentralCrossRef
14.
go back to reference Birken SA, Rohweder CL, Powell BJ, Shea CM, Scott J, Leeman J, et al. T-CaST: an implementation theory comparison and selection tool. Implement Sci. 2018;13(1):143.PubMedPubMedCentralCrossRef Birken SA, Rohweder CL, Powell BJ, Shea CM, Scott J, Leeman J, et al. T-CaST: an implementation theory comparison and selection tool. Implement Sci. 2018;13(1):143.PubMedPubMedCentralCrossRef
15.
go back to reference Strifler L, Barnsley JM, Hillmer M, Straus SE. Identifying and selecting implementation theories, models and frameworks: a qualitative study to inform the development of a decision support tool. BMC Med Inform Decis Mak. 2020;20(1):91.PubMedPubMedCentralCrossRef Strifler L, Barnsley JM, Hillmer M, Straus SE. Identifying and selecting implementation theories, models and frameworks: a qualitative study to inform the development of a decision support tool. BMC Med Inform Decis Mak. 2020;20(1):91.PubMedPubMedCentralCrossRef
16.
go back to reference Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated consolidated framework for implementation research based on user feedback. Implement Sci. 2022;17(1):75.PubMedPubMedCentralCrossRef Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated consolidated framework for implementation research based on user feedback. Implement Sci. 2022;17(1):75.PubMedPubMedCentralCrossRef
17.
go back to reference Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.PubMedPubMedCentralCrossRef Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.PubMedPubMedCentralCrossRef
18.
go back to reference Feldstein AC, Glasgow RE. A Practical, Robust Implementation and Sustainability Model (PRISM) for integrating research findings into practice. Jt Commun J Qual Patient Saf. 2008;34(4):228–43. Feldstein AC, Glasgow RE. A Practical, Robust Implementation and Sustainability Model (PRISM) for integrating research findings into practice. Jt Commun J Qual Patient Saf. 2008;34(4):228–43.
19.
go back to reference Green LW, Kreuter MW. Health program planning: an educational and ecological approach. 4th ed. New York: McGraw-Hill; 2005. Green LW, Kreuter MW. Health program planning: an educational and ecological approach. 4th ed. New York: McGraw-Hill; 2005.
20.
go back to reference Smith MY, Morrato E. Advancing the field of pharmaceutical risk minimization through application of implementation science best practices. Drug Saf. 2014;37(8):569–80.PubMedPubMedCentralCrossRef Smith MY, Morrato E. Advancing the field of pharmaceutical risk minimization through application of implementation science best practices. Drug Saf. 2014;37(8):569–80.PubMedPubMedCentralCrossRef
23.
go back to reference Ehlken B, Nishikawa C, Kaplan S, Dresco I, Granados D, Toussi M. Effectiveness of risk minimization measures for valproate: a drug utilization study based on implementation of a risk minimization programme in Europe, analysis of data from the UK. Curr Med Res Opin. 2022;38(3):461–8.PubMedCrossRef Ehlken B, Nishikawa C, Kaplan S, Dresco I, Granados D, Toussi M. Effectiveness of risk minimization measures for valproate: a drug utilization study based on implementation of a risk minimization programme in Europe, analysis of data from the UK. Curr Med Res Opin. 2022;38(3):461–8.PubMedCrossRef
24.
go back to reference Valproate banned without the pregnancy prevention programme [press release]. 2018. Valproate banned without the pregnancy prevention programme [press release]. 2018.
27.
go back to reference Bahri P, Morales DR, Inoubli A, Dogné JM, Straus S. Proposals for engaging patients and healthcare professionals in risk minimisation from an analysis of stakeholder input to the EU valproate assessment using the novel Analysing Stakeholder Safety Engagement Tool (ASSET). Drug Saf. 2021;44(2):193–209.PubMedCrossRef Bahri P, Morales DR, Inoubli A, Dogné JM, Straus S. Proposals for engaging patients and healthcare professionals in risk minimisation from an analysis of stakeholder input to the EU valproate assessment using the novel Analysing Stakeholder Safety Engagement Tool (ASSET). Drug Saf. 2021;44(2):193–209.PubMedCrossRef
28.
go back to reference Creswell J, Plano C, V L, Guttman M, Hanson W. Advanced mixed methods research designs. In: Tashakkori A, Teddlie C, editors. Handbook on mixed methods in the behavioral and social sciences. Thousand Oaks: Sage Publications; 2003. p. 209-40. Creswell J, Plano C, V L, Guttman M, Hanson W. Advanced mixed methods research designs. In: Tashakkori A, Teddlie C, editors. Handbook on mixed methods in the behavioral and social sciences. Thousand Oaks: Sage Publications; 2003. p. 209-40.
29.
go back to reference Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 2nd ed. Thousand Oaks: Sage Publications; 2011. Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 2nd ed. Thousand Oaks: Sage Publications; 2011.
31.
go back to reference Barbour RS. The case for combining qualitative and quantitative approaches in health services research. J Health Serv Res Policy. 1999;4(1):39–43.PubMedCrossRef Barbour RS. The case for combining qualitative and quantitative approaches in health services research. J Health Serv Res Policy. 1999;4(1):39–43.PubMedCrossRef
32.
go back to reference Artime E, Qizilbash N, Garrido-Estepa M, Vora P, Soriano-Gabarró M, Asiimwe A, Pocock S. Are risk minimization measures for approved drugs in Europe effective? A systematic review. Expert Opin Drug Saf. 2019;18(5):443–54.PubMedCrossRef Artime E, Qizilbash N, Garrido-Estepa M, Vora P, Soriano-Gabarró M, Asiimwe A, Pocock S. Are risk minimization measures for approved drugs in Europe effective? A systematic review. Expert Opin Drug Saf. 2019;18(5):443–54.PubMedCrossRef
33.
go back to reference Gridchyna I, Cloutier AM, Nkeng L, Craig C, Frise S, Moride Y. Methodological gaps in the assessment of risk minimization interventions: a systematic review. Pharmacoepidemiol Drug Saf. 2014;23(6):572–9.PubMedCrossRef Gridchyna I, Cloutier AM, Nkeng L, Craig C, Frise S, Moride Y. Methodological gaps in the assessment of risk minimization interventions: a systematic review. Pharmacoepidemiol Drug Saf. 2014;23(6):572–9.PubMedCrossRef
34.
go back to reference Grupstra RJ, Goedecke T, Scheffers J, Strassmann V, Gardarsdottir H. Review of studies evaluating effectiveness of risk minimization measures assessed by the European Medicines Agency between 2016 and 2021. Clin Pharmacol Ther. 2023;114(6):1285–92.PubMedCrossRef Grupstra RJ, Goedecke T, Scheffers J, Strassmann V, Gardarsdottir H. Review of studies evaluating effectiveness of risk minimization measures assessed by the European Medicines Agency between 2016 and 2021. Clin Pharmacol Ther. 2023;114(6):1285–92.PubMedCrossRef
36.
go back to reference Mazzaglia G, Straus SMJ, Arlett P, da Silva D, Janssen H, Raine J, Alteri E. Study design and evaluation of risk minimization measures: a review of studies submitted to the European Medicines Agency for cardiovascular, endocrinology, and metabolic drugs. Drug Saf. 2018;41(2):191–202.PubMedCrossRef Mazzaglia G, Straus SMJ, Arlett P, da Silva D, Janssen H, Raine J, Alteri E. Study design and evaluation of risk minimization measures: a review of studies submitted to the European Medicines Agency for cardiovascular, endocrinology, and metabolic drugs. Drug Saf. 2018;41(2):191–202.PubMedCrossRef
37.
go back to reference Okai J, Bahri P, Sarinic VM. Reporting quality of evaluations of additional risk minimisation measures by MAH: an assessment of studies submitted in the European Union. Drug Saf. 2021;44(12):1402–3. Okai J, Bahri P, Sarinic VM. Reporting quality of evaluations of additional risk minimisation measures by MAH: an assessment of studies submitted in the European Union. Drug Saf. 2021;44(12):1402–3.
38.
go back to reference Prieto L, Spooner A, Hidalgo-Simon A, Rubino A, Kurz X, Arlett P. Evaluation of the effectiveness of risk minimization measures. Pharmacoepidemiol Drug Saf. 2012;21(8):896–9.PubMedCrossRef Prieto L, Spooner A, Hidalgo-Simon A, Rubino A, Kurz X, Arlett P. Evaluation of the effectiveness of risk minimization measures. Pharmacoepidemiol Drug Saf. 2012;21(8):896–9.PubMedCrossRef
39.
go back to reference Russell AM, Morrato EH, Lovett RM, Smith MY. Quality of reporting on the evaluation of risk minimization programs: a systematic review. Drug Saf. 2020;43(5):427–46.PubMedCrossRef Russell AM, Morrato EH, Lovett RM, Smith MY. Quality of reporting on the evaluation of risk minimization programs: a systematic review. Drug Saf. 2020;43(5):427–46.PubMedCrossRef
40.
go back to reference Farcas A, Huruba M, Mogosan C. Study design, process and outcome indicators of post-authorization studies aimed at evaluating the effectiveness of risk minimization measures in the EU PAS Register. Br J Clin Pharmacol. 2019;85(3):476–91.PubMedCrossRef Farcas A, Huruba M, Mogosan C. Study design, process and outcome indicators of post-authorization studies aimed at evaluating the effectiveness of risk minimization measures in the EU PAS Register. Br J Clin Pharmacol. 2019;85(3):476–91.PubMedCrossRef
42.
go back to reference Toyserkani GA, Ewusie SB, Turk P, Quick J, Morrato EH. Evolution of cross-sectional survey protocol quality over time: a case series of index US REMS Knowledge Survey Protocols (2007–2020). Drug Saf. 2023;46(11):1073–87.PubMedCrossRef Toyserkani GA, Ewusie SB, Turk P, Quick J, Morrato EH. Evolution of cross-sectional survey protocol quality over time: a case series of index US REMS Knowledge Survey Protocols (2007–2020). Drug Saf. 2023;46(11):1073–87.PubMedCrossRef
43.
go back to reference Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019;7:64.PubMedPubMedCentralCrossRef Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019;7:64.PubMedPubMedCentralCrossRef
44.
go back to reference Rabin BA, Cakici J, Golden CA, Estabrooks PA, Glasgow RE, Gaglio B. A citation analysis and scoping systematic review of the operationalization of the Practical, Robust Implementation and Sustainability Model (PRISM). Implement Sci. 2022;17(1):62.PubMedPubMedCentralCrossRef Rabin BA, Cakici J, Golden CA, Estabrooks PA, Glasgow RE, Gaglio B. A citation analysis and scoping systematic review of the operationalization of the Practical, Robust Implementation and Sustainability Model (PRISM). Implement Sci. 2022;17(1):62.PubMedPubMedCentralCrossRef
45.
go back to reference Goedecke T, Morales DR, Pacurariu A, Kurz X. Measuring the impact of medicines regulatory interventions—systematic review and methodological considerations. Br J Clin Pharmacol. 2018;84(3):419–33.PubMedCrossRef Goedecke T, Morales DR, Pacurariu A, Kurz X. Measuring the impact of medicines regulatory interventions—systematic review and methodological considerations. Br J Clin Pharmacol. 2018;84(3):419–33.PubMedCrossRef
47.
go back to reference Haynes A, Loblay V. Rethinking barriers and enablers in qualitative health research: Limitations, alternatives and enhancements. Qualitative Health Research 2024;34(14):1371–83. Haynes A, Loblay V. Rethinking barriers and enablers in qualitative health research: Limitations, alternatives and enhancements. Qualitative Health Research 2024;34(14):1371–83.
48.
go back to reference Thompson W. Deprescribing: Moving beyond barriers and facilitators. Res in Soc Admin Pharm 2022;18(3):2547–9. Thompson W. Deprescribing: Moving beyond barriers and facilitators. Res in Soc Admin Pharm 2022;18(3):2547–9.
49.
go back to reference Wu J, Juhaeri J. The US Food and Drug Administration’s risk evaluation and mitigation strategy (REMS) program—current status and future direction. Clin Ther. 2016;38(12):2526–32.PubMedCrossRef Wu J, Juhaeri J. The US Food and Drug Administration’s risk evaluation and mitigation strategy (REMS) program—current status and future direction. Clin Ther. 2016;38(12):2526–32.PubMedCrossRef
50.
go back to reference Forman J, Heisler M, Damschroder LJ, Kaselitz E, Kerr EA. Development and application of the RE-AIM QuEST mixed methods framework for program evaluation. Prev Med Rep. 2017;6:322–8.PubMedPubMedCentralCrossRef Forman J, Heisler M, Damschroder LJ, Kaselitz E, Kerr EA. Development and application of the RE-AIM QuEST mixed methods framework for program evaluation. Prev Med Rep. 2017;6:322–8.PubMedPubMedCentralCrossRef
52.
go back to reference Smith MY, Morrato EH, Mora N, Nguyen V, Pinnock H, Winterstein AG. The reporting recommendations intended for pharmaceutical risk minimization evaluation studies: standards for reporting of implementation studies extension (RIMES-SE). Drug Saf. 2024;47(7):655–71.PubMedPubMedCentralCrossRef Smith MY, Morrato EH, Mora N, Nguyen V, Pinnock H, Winterstein AG. The reporting recommendations intended for pharmaceutical risk minimization evaluation studies: standards for reporting of implementation studies extension (RIMES-SE). Drug Saf. 2024;47(7):655–71.PubMedPubMedCentralCrossRef
53.
go back to reference Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for reporting implementation studies (StaRI) statement. BMJ. 2017;356: i6795.PubMedPubMedCentralCrossRef Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for reporting implementation studies (StaRI) statement. BMJ. 2017;356: i6795.PubMedPubMedCentralCrossRef
54.
go back to reference Bahri P, Genov G, Arlett P, Šarinić VM, Korakianiti E, Nolte A, et al. The STAR Compass to guide future pharmacovigilance based on a 10-year review of the strengthened EU system. Drug Saf. 2024;47:941–56.PubMedPubMedCentralCrossRef Bahri P, Genov G, Arlett P, Šarinić VM, Korakianiti E, Nolte A, et al. The STAR Compass to guide future pharmacovigilance based on a 10-year review of the strengthened EU system. Drug Saf. 2024;47:941–56.PubMedPubMedCentralCrossRef
Metadata
Title
Using Mixed Methods to Evaluate Risk Minimisation Programs in Europe and the USA: An Innovative Blueprint
Authors
Meredith Y. Smith
Rachel Davis
Priya Bahri
Delphine Saragoussi
Viviana Nguyen
Gita A. Toyserkani
Alison Hamilton
Publication date
12-03-2025
Publisher
Springer International Publishing
Published in
Drug Safety
Print ISSN: 0114-5916
Electronic ISSN: 1179-1942
DOI
https://doi.org/10.1007/s40264-025-01533-w