Skip to main content
Top
Published in: PharmacoEconomics 11/2019

01-11-2019 | Practical Application

A Need for Change! A Coding Framework for Improving Transparency in Decision Modeling

Authors: Fernando Alarid-Escudero, Eline M. Krijkamp, Petros Pechlivanoglou, Hawre Jalal, Szu-Yu Zoe Kao, Alan Yang, Eva A. Enns

Published in: PharmacoEconomics | Issue 11/2019

Login to get access

Abstract

The use of open-source programming languages, such as R, in health decision sciences is growing and has the potential to facilitate model transparency, reproducibility, and shareability. However, realizing this potential can be challenging. Models are complex and primarily built to answer a research question, with model sharing and transparency relegated to being secondary goals. Consequently, code is often neither well documented nor systematically organized in a comprehensible and shareable approach. Moreover, many decision modelers are not formally trained in computer programming and may lack good coding practices, further compounding the problem of model transparency. To address these challenges, we propose a high-level framework for model-based decision and cost-effectiveness analyses (CEA) in R. The proposed framework consists of a conceptual, modular structure and coding recommendations for the implementation of model-based decision analyses in R. This framework defines a set of common decision model elements divided into five components: (1) model inputs, (2) decision model implementation, (3) model calibration, (4) model validation, and (5) analysis. The first four components form the model development phase. The analysis component is the application of the fully developed decision model to answer the policy or the research question of interest, assess decision uncertainty, and/or to determine the value of future research through value of information (VOI) analysis. In this framework, we also make recommendations for good coding practices specific to decision modeling, such as file organization and variable naming conventions. We showcase the framework through a fully functional, testbed decision model, which is hosted on GitHub for free download and easy adaptation to other applications. The use of this framework in decision modeling will improve code readability and model sharing, paving the way to an ideal, open-source world.
Appendix
Available only for authorised users
Literature
1.
go back to reference Taichman DB. Data sharing statements for clinical trials: a requirement of the international committee of medical journal editors. Ann Intern Med. 2017;14:e1002315–e1002315. Taichman DB. Data sharing statements for clinical trials: a requirement of the international committee of medical journal editors. Ann Intern Med. 2017;14:e1002315–e1002315.
3.
go back to reference Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB. Model transparency and validation: a report of the ISPOR-SMDM modeling good research practices task force-7. Med Decis Mak. 2012;32:733–43.CrossRef Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB. Model transparency and validation: a report of the ISPOR-SMDM modeling good research practices task force-7. Med Decis Mak. 2012;32:733–43.CrossRef
4.
go back to reference Cohen JT, Neumann PJ, Wong JB. A call for open-source cost-effectiveness analysis. Ann Intern Med. 2017;167(6):432–3.CrossRef Cohen JT, Neumann PJ, Wong JB. A call for open-source cost-effectiveness analysis. Ann Intern Med. 2017;167(6):432–3.CrossRef
5.
go back to reference Baio G, Heath A. When simple becomes complicated: why Excel should lose its place at the top table. Glob Reg Heal Technol Assess. 2017;4(1):e3–6. Baio G, Heath A. When simple becomes complicated: why Excel should lose its place at the top table. Glob Reg Heal Technol Assess. 2017;4(1):e3–6.
8.
go back to reference Dunlop WCN, Mason N, Kenworthy J, Akehurst RL. Benefits, challenges and potential strategies of open source health economic models. Pharmacoeconomics. 2017;35:125–8.CrossRef Dunlop WCN, Mason N, Kenworthy J, Akehurst RL. Benefits, challenges and potential strategies of open source health economic models. Pharmacoeconomics. 2017;35:125–8.CrossRef
9.
go back to reference Sampson CJ, Wrightson T. Model registration: a call to action. Pharmacoecon Open. 2017;1:73–7.CrossRef Sampson CJ, Wrightson T. Model registration: a call to action. Pharmacoecon Open. 2017;1:73–7.CrossRef
11.
go back to reference Jalal H, Pechlivanoglou P, Krijkamp E, Alarid-Escudero F, Enns EA, Hunink MGM. An overview of R in health decision sciences. Med Decis Mak. 2017;37:735–46.CrossRef Jalal H, Pechlivanoglou P, Krijkamp E, Alarid-Escudero F, Enns EA, Hunink MGM. An overview of R in health decision sciences. Med Decis Mak. 2017;37:735–46.CrossRef
12.
go back to reference Decision Analysis in R for Technologies in Health (DARTH) workgroup. Decision analysis in R for technologies in health [Internet]. 2019. http://darthworkgroup.com. Accessed 1 Jan 2019. Decision Analysis in R for Technologies in Health (DARTH) workgroup. Decision analysis in R for technologies in health [Internet]. 2019. http://​darthworkgroup.​com. Accessed 1 Jan 2019.
13.
go back to reference R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2019. R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2019.
14.
go back to reference Marwick B, Boettiger C, Mullen L. Packaging data analytical work reproducibly using R (and friends). Am Stat. 2018;72:80–8.CrossRef Marwick B, Boettiger C, Mullen L. Packaging data analytical work reproducibly using R (and friends). Am Stat. 2018;72:80–8.CrossRef
15.
go back to reference Stout NK, Knudsen AB, Kong CY, Mcmahon PM, Gazelle GS. Calibration methods used in cancer simulation models and suggested reporting guidelines. Pharmacoeconomics. 2009;27:533–45.CrossRef Stout NK, Knudsen AB, Kong CY, Mcmahon PM, Gazelle GS. Calibration methods used in cancer simulation models and suggested reporting guidelines. Pharmacoeconomics. 2009;27:533–45.CrossRef
16.
go back to reference Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM modeling good research practices task force working group-6. Med Decis Mak. 2012;32:722–32.CrossRef Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM modeling good research practices task force working group-6. Med Decis Mak. 2012;32:722–32.CrossRef
17.
go back to reference Alarid-Escudero F, MacLehose RF, Peralta Y, Kuntz KM, Enns EA. Nonidentifiability in model calibration and implications for medical decision making. Med Decis Mak. 2018;38:810–21.CrossRef Alarid-Escudero F, MacLehose RF, Peralta Y, Kuntz KM, Enns EA. Nonidentifiability in model calibration and implications for medical decision making. Med Decis Mak. 2018;38:810–21.CrossRef
18.
go back to reference Sargent RG. Verification and validation of simulation models. J Simul. 2013;7:12–24.CrossRef Sargent RG. Verification and validation of simulation models. J Simul. 2013;7:12–24.CrossRef
19.
go back to reference Goldhaber-Fiebert JD, Stout NK, Goldie SJ. Empirically evaluating decision-analytic models. Value Health. 2010;13:667–74.CrossRef Goldhaber-Fiebert JD, Stout NK, Goldie SJ. Empirically evaluating decision-analytic models. Value Health. 2010;13:667–74.CrossRef
20.
go back to reference Rutter CM, Savarino JE. An evidence-based microsimulation model for colorectal cancer: validation and application. Cancer Epidemiol Biomark Prev. 2010;19:1992–2002.CrossRef Rutter CM, Savarino JE. An evidence-based microsimulation model for colorectal cancer: validation and application. Cancer Epidemiol Biomark Prev. 2010;19:1992–2002.CrossRef
21.
go back to reference Rutter CM, Knudsen AB, Marsh TL, Doria-Rose VP, Johnson E, Pabiniak C, et al. Validation of models used to inform colorectal cancer screening guidelines: accuracy and implications. Med Decis Mak. 2016;36:604–14.CrossRef Rutter CM, Knudsen AB, Marsh TL, Doria-Rose VP, Johnson E, Pabiniak C, et al. Validation of models used to inform colorectal cancer screening guidelines: accuracy and implications. Med Decis Mak. 2016;36:604–14.CrossRef
22.
go back to reference Kopec JA, Finès P, Manuel DG, Buckeridge DL, Flanagan WM, Oderkirk J, et al. Validation of population-based disease simulation models: a review of concepts and methods. BMC Public Health. 2010;10:710.CrossRef Kopec JA, Finès P, Manuel DG, Buckeridge DL, Flanagan WM, Oderkirk J, et al. Validation of population-based disease simulation models: a review of concepts and methods. BMC Public Health. 2010;10:710.CrossRef
24.
go back to reference Krijkamp EM, Alarid-Escudero F, Enns E, Pechlivanoglou P, Hunink MM, Jalal H. A multidimensional array representation of state-transition model dynamics. bioRxiv 670612. 2019. Krijkamp EM, Alarid-Escudero F, Enns E, Pechlivanoglou P, Hunink MM, Jalal H. A multidimensional array representation of state-transition model dynamics. bioRxiv 670612. 2019.
25.
go back to reference Sculpher MJ, Basu A, Kuntz KM, Meltzer DO. Reflecting uncertainty in cost-effectiveness analysis. In: Neumann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG, editors. Cost-effectiveness heal med. 2nd ed. New York: Oxford University Press; 2017. p. 289–318. Sculpher MJ, Basu A, Kuntz KM, Meltzer DO. Reflecting uncertainty in cost-effectiveness analysis. In: Neumann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG, editors. Cost-effectiveness heal med. 2nd ed. New York: Oxford University Press; 2017. p. 289–318.
26.
go back to reference Alarid-Escudero F, Enns EA, Kuntz KM, Michaud TL, Jalal H. “Time traveling is just too dangerous” but some methods are worth revisiting: the advantages of expected loss curves over cost-effectiveness acceptability curves and frontier. Value Health. 2019;22:611–8.CrossRef Alarid-Escudero F, Enns EA, Kuntz KM, Michaud TL, Jalal H. “Time traveling is just too dangerous” but some methods are worth revisiting: the advantages of expected loss curves over cost-effectiveness acceptability curves and frontier. Value Health. 2019;22:611–8.CrossRef
27.
go back to reference Raiffa H, Schlaifer RO. Applied statistical decision theory. Cambridge: Harvard Business School; 1961. Raiffa H, Schlaifer RO. Applied statistical decision theory. Cambridge: Harvard Business School; 1961.
28.
go back to reference Claxton K, Posnett J. An economic approach to clinical trial design and research priority-setting. Health Econ. 1996;5:513–24.CrossRef Claxton K, Posnett J. An economic approach to clinical trial design and research priority-setting. Health Econ. 1996;5:513–24.CrossRef
29.
go back to reference Jutkowitz E, Alarid-Escudero F, Kuntz KM, Jalal H. The curve of optimal sample size (COSS): a graphical representation of the optimal sample size from a value of information analysis. Pharmacoeconomics. 2019;37:871–7.CrossRef Jutkowitz E, Alarid-Escudero F, Kuntz KM, Jalal H. The curve of optimal sample size (COSS): a graphical representation of the optimal sample size from a value of information analysis. Pharmacoeconomics. 2019;37:871–7.CrossRef
30.
go back to reference Wickham H. R packages: organize, test, document, and share your code. Spencer A, Marie Beaugureau, editors. Sebastopol: O’Reilly Media; 2015. Wickham H. R packages: organize, test, document, and share your code. Spencer A, Marie Beaugureau, editors. Sebastopol: O’Reilly Media; 2015.
31.
go back to reference Cooper N, Hsing P-Y, editors. A guide to reproducible code in ecology and evolution. London: British Ecology Society; 2017. Cooper N, Hsing P-Y, editors. A guide to reproducible code in ecology and evolution. London: British Ecology Society; 2017.
32.
go back to reference Kleijnen JPC. Verification and validation of simulation models. Eur J Oper Res. 1995;82:145–62.CrossRef Kleijnen JPC. Verification and validation of simulation models. Eur J Oper Res. 1995;82:145–62.CrossRef
35.
go back to reference Martin RC. Clean code: a handbook of agile software craftsmanship. Boston: Pearson Education; 2009. Martin RC. Clean code: a handbook of agile software craftsmanship. Boston: Pearson Education; 2009.
37.
go back to reference Beeley C. Web application development with R using Shiny. Birmingham: Packt Publishing Ltd; 2013. Beeley C. Web application development with R using Shiny. Birmingham: Packt Publishing Ltd; 2013.
38.
go back to reference Incerti D, Curtis JR, Shafrin J, Lakdawalla DN, Jansen JP. A flexible open-source decision model for value assessment of biologic treatment for rheumatoid arthritis. Pharmacoeconomics. 2019;37:829–43.CrossRef Incerti D, Curtis JR, Shafrin J, Lakdawalla DN, Jansen JP. A flexible open-source decision model for value assessment of biologic treatment for rheumatoid arthritis. Pharmacoeconomics. 2019;37:829–43.CrossRef
39.
go back to reference Xie Y. Bookdown: authoring books with R Markdown. Boca Raton, FL: CRC Press; 2016.CrossRef Xie Y. Bookdown: authoring books with R Markdown. Boca Raton, FL: CRC Press; 2016.CrossRef
40.
go back to reference Enns EA, Cipriano LE, Simons CT, Kong CY. Identifying best-fitting inputs in health-economic model calibration: a pareto frontier approach. Med Decis Mak. 2015;35:170–82.CrossRef Enns EA, Cipriano LE, Simons CT, Kong CY. Identifying best-fitting inputs in health-economic model calibration: a pareto frontier approach. Med Decis Mak. 2015;35:170–82.CrossRef
42.
go back to reference David O, Ascough JC, Lloyd W, Green TR, Rojas KW, Leavesley GH, et al. A software engineering perspective on environmental modeling framework design: the object modeling system. Environ Model Softw. 2013;39:201–13.CrossRef David O, Ascough JC, Lloyd W, Green TR, Rojas KW, Leavesley GH, et al. A software engineering perspective on environmental modeling framework design: the object modeling system. Environ Model Softw. 2013;39:201–13.CrossRef
Metadata
Title
A Need for Change! A Coding Framework for Improving Transparency in Decision Modeling
Authors
Fernando Alarid-Escudero
Eline M. Krijkamp
Petros Pechlivanoglou
Hawre Jalal
Szu-Yu Zoe Kao
Alan Yang
Eva A. Enns
Publication date
01-11-2019
Publisher
Springer International Publishing
Published in
PharmacoEconomics / Issue 11/2019
Print ISSN: 1170-7690
Electronic ISSN: 1179-2027
DOI
https://doi.org/10.1007/s40273-019-00837-x

Other articles of this Issue 11/2019

PharmacoEconomics 11/2019 Go to the issue