Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
The European Journal of Health Economics
Published in: The European Journal of Health Economics 6/2019

Open Access 01-08-2019 | Original Paper

Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art

Authors: Mónica D. Oliveira, Inês Mataloto, Panos Kanavos

Published in: The European Journal of Health Economics | Issue 6/2019

Login to get access

Abstract

Background

Multi-criteria decision analysis (MCDA) concepts, models and tools have been used increasingly in health technology assessment (HTA), with several studies pointing out practical and theoretical issues related to its use. This study provides a critical review of published studies on MCDA in the context of HTA by assessing their methodological quality and summarising methodological challenges.

Methods

A systematic review was conducted to identify studies discussing, developing or reviewing the use of MCDA in HTA using aggregation approaches. Studies were classified according to publication time and type, country of study, technology type and study type. The PROACTIVE-S approach was constructed and used to analyse methodological quality. Challenges and limitations reported in eligible studies were collected and summarised; this was followed by a critical discussion on research requirements to address the identified challenges.

Results

129 journal articles were eligible for review, 56% of which were published in 2015–2017; 42% focused on pharmaceuticals; 36, 26 and 18% reported model applications, issues regarding MCDA implementation analyses, and proposing frameworks, respectively. Poor compliance with good methodological practice (< 25% complying studies) was found regarding behavioural analyses, discussion of model assumptions and uncertainties, modelling of value functions, and dealing with judgment inconsistencies. The five most reported challenges related to evidence and data synthesis; value system differences and participant selection issues; participant difficulties; methodological complexity and resource balance; and criteria and attributes modelling. A critical discussion on ways to address these challenges ensues.

Discussion

Results highlight the need for advancement in robust methodologies, procedures and tools to improve methodological quality of MCDA in HTA studies. Research pathways include developing new model features, good practice guidelines, technologies to enable participation and behavioural research.
Appendix
Available only for authorised users
Literature
1.
Velasco-Garrido M., Busse R.: Health Technology Assessment: An Introduction to Objectives, Role of Evidence, and Structure in Europe, in Policy Brief, World Health Organization and European Observatory on Health Systems and Policies, Editors. European Observatory on Health Systems and Policies: Copenhagen (2005)
2.
Sorenson C., Drummond M., Kanavos P.: Ensuring value for money in health care: the role of health technology assessment in the European Union. World Health Organization and European Observatory on Health Care Systems (2008)
3.
4.
Drummond, M., Tarricone, R., Torbica, A.: Assessing the added value of health technologies: reconciling different perspectives. Value Health 16(S1), 7–13 (2013)CrossRef
5.
6.
Sorenson C, et al. How Can the Impact of Health Technology Assessments be Enhanced?, in Health Systems and Policy Analysis Briefings, WHO Regional Office for Europe and European Observatory on Health Systems and Policies, Editor. World Health Organization and European Observatory on Health Systems and Policies (2008)
7.
Ciani, O., et al.: Linking the regulatory and reimbursement processes for medical devices: the need for integrated assessments. Health Econ 26(Suppl 1), 13–29 (2017)CrossRefPubMed
8.
Sampietro-Colom, L., Martin, J.: Hospital-based health technology assessment: the next frontier. In: Sampietro-Colom, L., Martin, J. (eds.) Hospital-Based Health Technology Assessment: The Next Frontier for Health Technology Assessment. Springer, Switzerland (2016)CrossRef
9.
Tarricone, R., Torbica, A., Drummond, M.: Challenges in the assessment of medical devices: the MedtecHTA project. Health Econ 26(Suppl 1), 5–12 (2017)CrossRefPubMed
10.
Hubbard, D.W.: How to Measure Anything: Finding the Value of Intangibles in Business, vol. 3, p. 432. Wiley, New Jersey (2014)
11.
Devlin, N.J., Sussex, J.: Incorporating Multiple Criteria in HTA: Methods and Processes, p. 60. Office of Health Economics, London (2011)
12.
Angelis, A., Lange, A., Kanavos, P.: Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries. Eur J Health Econ 19, 123–152 (2018)CrossRefPubMed
13.
Nicod, E.: Why do health technology assessment coverage recommendations for the same drugs differ across settings? Applying a mixed methods framework to systematically compare orphan drug decisions in four European countries. Eur J Health Econ 18(6), 715–730 (2017)CrossRefPubMed
14.
Simon, H.A.: Models of Man: Social and Rational. Wiley, New York (1957)
15.
16.
Keeney, R., Raiffa, H.: Decisions with Multiple Objectives: Preferences and Value Tradeoffs. Wiley, Cambridge (1976)
17.
Howard, R.A.: Speaking of decisions: precise decision language. Decis Anal 1(2), 71–78 (2004)CrossRef
18.
Thokala, P., et al.: Multiple criteria decision analysis for health care decision making-an introduction: report 1 of the ISPOR MCDA emerging good practices task force. Value Health 19(1), 1–13 (2016)CrossRefPubMed
19.
Marsh, K., et al.: Multiple criteria decision analysis for health care decision making-emerging good practices: report 2 of the ISPOR MCDA emerging good practices task force. Value Health 19(2), 125–137 (2016)CrossRefPubMed
20.
Diaby, V., Campbell, K., Goeree, R.: Multi-criteria decision analysis (MCDA) in health care: a bibliometric analysis. Oper Res Health Care 2(1–2), 20–24 (2013)CrossRef
21.
Marsh, K., et al.: Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature. PharmacoEconomics 32(4), 345–365 (2014)CrossRefPubMed
22.
23.
Adunlin, G., Diaby, V., Xiao, H.: Application of multicriteria decision analysis in health care: a systematic review and bibliometric analysis. Health Expect 18(6), 1894–1905 (2015)CrossRefPubMed
24.
Thokala, P., Duenas, A.: Multiple criteria decision analysis for health technology assessment. Value Health 15(8), 1172–1181 (2012)CrossRefPubMed
25.
Sussex, J., Towse, A., Devlin, N.: Operationalizing value-based pricing of medicines a taxonomy of approaches. PharmacoEconomics 31(1), 1–10 (2013)CrossRefPubMed
26.
Angelis, A., Kanavos, P.: Value-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessment. PharmacoEconomics 34(5), 435–446 (2016)CrossRefPubMedPubMedCentral
27.
Goetghebeur, M.M., et al.: Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): applying the EVIDEM framework to medicines appraisal. Med Decis Mak 32(2), 376 (2012)CrossRef
28.
Baltussen, R., et al.: Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health 20, 256–260 (2017)CrossRefPubMed
29.
Wiseman, V., et al.: Using economic evidence to set healthcare priorities in low-income and lower-middle-income countries: a systematic review of methodological frameworks. Health Econ 25(S1), 140–161 (2016)CrossRefPubMedPubMedCentral
30.
Angelis, A., Kanavos, P.: Multiple criteria decision analysis (MCDA) for evaluating new medicines in health technology assessment and beyond: the advance value framework. Soc Sci Med 188, 137–156 (2017)CrossRefPubMed
31.
Garcia-Hernandez, A.: A note on the validity and reliability of multi-criteria decision analysis for the benefit-risk assessment of medicines. Drug Saf 38(11), 1049–1057 (2015)CrossRefPubMed
32.
Morton, A.: Treacle and smallpox: two tests for multicriteria decision analysis models in health technology assessment. Value Health 20, 512–515 (2017)CrossRefPubMed
33.
Antioch, K.M., et al.: International lessons in new methods for grading and integrating cost effectiveness evidence into clinical practice guidelines. Cost Eff Resour Alloc 15, 1–15 (2017)CrossRefPubMedPubMedCentral
34.
Keeney, R.L.: Value-focused Thinking: A Path to Creative Decision making. Harvard University Press, Harvard (1992)
35.
36.
Marsh, K., et al.: Amplifying each patient’s voice: a systematic review of multi-criteria decision analyses involving patients. Appl Health Econ Health Pol 15(2), 155–162 (2017)CrossRef
37.
Hunink, M.G.M., et al.: Decision Making in Health and Medicine: Integrating Evidence and Values, vol. 2, p. 446. Cambridge University Press, Cambridge (2014)CrossRef
38.
Hammond, J.S., Keeney, R.L., Raiffa, H.: Smart Choices: A Practical Guide to Making Better Decisions. Harvard Business School Press, Boston (1999)
39.
Von Winterfeldt, D., Edwards, W.: Decision Analysis and Behavioral Research. University Press, Cambridge (1986)
40.
Belton, V., Stewart, T.J.: Multiple Criteria Decision Analysis: An Integrated Approach. Springer, US (2002)CrossRef
41.
Bana-e-Costa, C.A., Mateus, R.: Common critical mistakes in evaluations with multiple criteria. In: Santos, A., Cardadeiro, E., Verga Matos, P. (eds.) Estudos de Homenagem ao Professor José Amado da Silva, pp. 277–316. Sílabas & Desafios, Lisboa (2017)
42.
Tervonen, T., et al.: Applying multiple criteria decision analysis to comparative benefit-risk assessment: choosing among statins in primary prevention. Med Decis Mak 35(7), 859–871 (2015)CrossRef
43.
Muhlbacher, A.C., Kaczynski, A.: Making good decisions in healthcare with multi-criteria decision analysis: the use, current research and future development of MCDA. Appl Health Econ Health Pol 14(1), 29–40 (2016)CrossRef
44.
Phillips, L.: Best practice for MCDA in healthcare. In: Marsh, K. (ed.) Multi-Criteria Decision Analysis to Support Healthcare Decisions, pp. 305–323. Springer, Cham (2017)
45.
Phillips, L.D., Bana-e-Costa, C.A.: Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing. Ann Oper Res 154(1), 51–68 (2007)CrossRef
46.
Cherns, A.: Principles of socio-technical design. Human Relat 29, 783–792 (1976)CrossRef
47.
Bana-e-Costa C.A., Beinat E.: Model-structuring in public decision-aiding. Operational Research working papers (LSEOR 05.79). Operational Research Group, Department of Management, London School of Economics and Political Science, London. ISBN 0753017016 (2005)
48.
Bana-e-Costa, C.A., et al.: Development of reusable bid evaluation models for the Portuguese Electric Transmission Company. Dec Anal 5(1), 22–42 (2008)CrossRef
49.
Phillips, L.: A theory of requisite decision models. Acta Physiol (Oxf) 56(1–3), 29–48 (1984)
50.
Hughes, D., et al.: Recommendations for benefit–risk assessment methodologies and visual representations. Pharmacoepidemiol Drug Saf 25(3), 251–262 (2016)CrossRefPubMed
51.
Phillips, L.: Best practice for MCDA in healthcare. In: Marsh, K. (ed.) Multi-Criteria Decision Analysis to Support Healthcare Decisions, pp. 311–329. Springer International Publishing, Cham (2007)
52.
Bana-e-Costa, C.A., et al.: A socio-technical approach for group decision support in public strategic planning: the Pernambuco PPA case. Group Decis Negot 23(1), 5–29 (2014)CrossRef
53.
Baron, J.: Thinking and Deciding, 4th edn. Cambridge University Press, New York (2008)
54.
Fasolo, B., Bana-e-Costa, C.A.: Tailoring value elicitation to decision makers’ numeracy and fluency: expressing value judgments in numbers or words. OMEGA 44, 83–90 (2014)CrossRef
55.
Montibeller, G., von Winterfeldt, D.: Cognitive and motivational biases in decision and risk analysis. Risk Anal 35(7), 1230–1251 (2015)CrossRefPubMed
56.
Gilabert-Perramon, A., et al.: Drug evaluation and decision making in Catalonia: development and validation of a methodological framework based on multi-criteria decision analysis (MCDA) for orphan drugs. Int J Technol Assess Health Care 33(1), 111–120 (2017)CrossRefPubMed
57.
Iskrov, G., Miteva-Katrandzhieva, T., Stefanov, R.: Multi-criteria decision analysis for assessment and appraisal of orphan drugs. Front Public Health 4, 214 (2016)CrossRefPubMedPubMedCentral
58.
Kolasa, K., et al.: Potential impact of the implementation of multiple-criteria decision analysis (MCDA) on the Polish pricing and reimbursement process of orphan drugs. Orphanet J Rare Dis 11, 1–12 (2016)CrossRef
59.
60.
Sussex, J., et al.: A pilot study of multicriteria decision analysis for valuing orphan medicines. Value Health 16(8), 1163–1169 (2013)CrossRefPubMed
61.
Angelis, A., et al.: Multiple criteria decision analysis in the context of health technology assessment: a simulation exercise on metastatic colorectal cancer with multiple stakeholders in the English setting. BMC Med Inform Decis Mak 17(1), 149 (2017)CrossRefPubMedPubMedCentral
62.
Kwon, S.H., et al.: Eliciting societal preferences of reimbursement decision criteria for anti cancer drugs in South Korea. Expert Rev Pharmacoecon Outcomes Res 17(4), 411–419 (2017)CrossRefPubMed
63.
64.
Danner, M., et al.: Integrating patients’ views into health technology assessment: analytic hierarchy process (AHP) as a method to elicit patient preferences. Int J Technol Assess Health Care 27(4), 369–375 (2011)CrossRefPubMed
65.
Hummel, M.J.M., et al.: Using the analytic hierarchy process to elicit patient preferences: prioritizing multiple outcome measures of antidepressant drug treatment. Patient: Patient-Cent Outcomes Res 5(4), 225–237 (2012)CrossRef
66.
Hsu, J.C., et al.: Net Clinical benefit of oral anticoagulants: a multiple criteria decision analysis. PLoS One 10(4), 1–15 (2015)
67.
Tervonen, T., et al.: Comparison of oral anticoagulants for stroke prevention in nonvalvular atrial fibrillation: a multicriteria decision analysis. Value Health 20(10), 1394–1402 (2017)CrossRefPubMed
68.
Moore, A., et al.: Use of multicriteria decision analysis for assessing the benefit and risk of over-the-counter analgesics. J Pharm Pharmacol 69(10), 1364–1373 (2017)CrossRefPubMed
69.
70.
Danner, M., et al.: How well can Analytic Hierarchy Process be used to elicit individual preferences? Insights from a survey in patients suffering from age-related macular degeneration. Patient: Patient-Cent Outcomes Res 9(5), 481–492 (2016)CrossRef
71.
de Greef-van der Sandt, I., et al.: A quantitative benefit-risk assessment approach to improve decision making in drug development: Application of a multicriteria decision analysis model in the development of combination therapy for overactive bladder. Clin Pharmacol Ther 99(4), 442 (2016)CrossRefPubMed
72.
Felli, J.C., Noel, R.A., Cavazzoni, P.A.: A multiattribute model for evaluating the benefit-risk profiles of treatment alternatives. Med Decis Mak 29(1), 104 (2009)CrossRef
73.
Goetghebeur, M.M., et al.: Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decisionmaking framework to growth hormone for Turner syndrome patients. Cost Eff Resour Alloc 8, 4–18 (2010)CrossRefPubMedPubMedCentral
74.
Goetghebeur, M.M., et al.: Pragmatic multicriteria decision analysis (MCDA) combined with advanced pharmacoepidemiology for benefit-risk assessments of medicines adapted to the real-life constraints of regulators: development and case study. Ther Innov Regul Sci 50(5), 620–631 (2016)PubMed
75.
Hoshikawa, K., Ono, S.: Discrepancies between multicriteria decision analysis-based ranking and intuitive ranking for pharmaceutical benefit-risk profiles in a hypothetical setting. J Clin Pharm Ther 42(1), 80–86 (2017)CrossRefPubMed
76.
Marsh, K., et al.: Evaluation of COPD treatments: a multicriteria decision analysis of aclidinium and tiotropium in the United States. Value Health 20(1), 132–140 (2017)CrossRefPubMed
77.
Nixon, R., et al.: A case study using the PrOACT-URL and BRAT frameworks for structured benefit risk assessment. Biom J 58(1), 8 (2016)CrossRefPubMed
78.
Jaramillo, H.E.C., Goetghebeur, M., Moreno-Mattar, O.: Testing multi-criteria decision analysis for more transparent resource-allocation decision making in Colombia. Int J Technol Assess Health Care 32(4), 307 (2016)CrossRef
79.
Kulak, O., Goren, H.G., Supciller, A.A.: A new multi criteria decision making approach for medical imaging systems considering risk factors. Appl Soft Comput 35, 931–941 (2015)CrossRef
80.
Ivlev, I., Vacek, J., Kneppo, P.: Multi-criteria decision analysis for supporting the selection of medical devices under uncertainty. Eur J Oper Res 247(1), 216–228 (2015)CrossRef
81.
Churilov, L., et al.: Multiattribute selection of acute stroke imaging software platform for extending the time for thrombolysis in emergency neurological deficits (EXTEND) clinical trial. Int J Stroke 8(3), 204–210 (2013)CrossRefPubMed
82.
Hilgerink, M.P., et al.: Assessment of the added value of the Twente Photoacoustic Mammoscope in breast cancer diagnosis. Med Devices (Auckl) 4, 107–115 (2011)
83.
Diaz-Ledezma, C., Parvizi, J.: Surgical approaches for cam femoroacetabular impingement: the use of multicriteria decision analysis. Clin Orthop Relat Res 471(8), 2509–2516 (2013)CrossRefPubMedPubMedCentral
84.
Hummel, J.M., et al.: Predicting the health economic performance of new non-fusion surgery in adolescent idiopathic scoliosis. J Orthop Res 30(9), 1453–1458 (2012)CrossRefPubMed
85.
Hummel, J.M., et al.: Supporting the project portfolio selection decision of research and development investments by means of multi-criteria resource allocation modelling. In: Marsh, K., et al. (eds.) Multi-Criteria Decision Analysis to Support Healthcare Decisions, pp. 89–103. Springer, Cham (2017)CrossRef
86.
Ritrovato, M., et al.: Decision-oriented health technology assessment: one step forward in supporting the decision-making process in hospitals. Value Health 18(4), 505–511 (2015)CrossRefPubMed
87.
Ottardi, C., et al.: A comparative analysis of a disposable and a reusable pedicle screw instrument kit for lumbar arthrodesis: integrating HTA and MCDA. Health Econ Rev 7(1), 17 (2017)CrossRefPubMedPubMedCentral
88.
89.
Martelli, N., et al.: Combining multi-criteria decision analysis and mini-health technology assessment: a funding decision-support tool for medical devices in a university hospital setting. J Biomed Inform 59, 201–208 (2016)CrossRefPubMed
90.
Miot, J., et al.: Field testing of a multicriteria decision analysis (MCDA) framework for coverage of a screening test for cervical cancer in South Africa. Cost Eff Resour Alloc 10(1), 2 (2012)CrossRefPubMedPubMedCentral
91.
Airoldi, M., Morton, A.: Portfolio decision analysis for population health. In: Salo, A., Keisler, J., Morton, A. (eds.) Portfolio Decision Analysis: Improved Methods for Resource Allocation, pp. 359–381. Springer, New York (2011)CrossRef
92.
Airoldi, M., et al.: STAR–people-powered prioritization: a 21st-century solution to allocation headaches. Med Decis Mak 34(8), 965–975 (2014)CrossRef
93.
94.
Oliveira, M.D., et al.: Prioritizing health care interventions: a multicriteria resource allocation model to inform the choice of community care programmes. In: Tànfani, E., Testi, A. (eds.) Advanced Decision Making Methods Applied to Health Care, pp. 141–154. Milano, Springer Milan (2012)CrossRef
95.
Le Gales, C., Moatti, J.P.: Searching for consensus through multi-criteria decision analysis: assessment of screening strategies for hemoglobinopathies in southeastern France. Int J Technol Assess Health Care 6(3), 430–449 (1990)CrossRefPubMed
96.
Peacock, S.J., et al.: Priority setting in health care using multi-attribute utility theory and programme budgeting and marginal analysis (PBMA). Soc Sci Med 64(4), 897–910 (2007)CrossRefPubMed
97.
Reddy, B., Thokala, P., Duenas, A.: Mcda for resource allocation at a local level: an application in the UK. In: Marsh, K., et al. (eds.) Multi-Criteria Decision Analysis to Support Healthcare Decisions, pp. 175–198. Springer, Cham (2017)CrossRef
98.
Bots, P.W.G., Hulshof, J.A.M.: Designing multi-criteria decision analysis processes for priority setting in health policy. J Multi Criteria Decis Anal 9(1–3), 56–75 (2000)CrossRef
99.
Youngkong, S., et al.: Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in thailand. Value Health 15(6), 961–970 (2012)CrossRefPubMed
100.
Mobinizadeh, M., et al.: A model for priority setting of health technology assessment: the experience of AHP-TOPSIS combination approach. DARU 24, 1–12 (2016)CrossRef
101.
Walker, S., et al.: A universal framework for the benefit-risk assessment of medicines: is this the way forward? Ther Innov Regul Sci 49(1), 17–25 (2015)PubMed
102.
Baeten, S.A., et al.: Incorporating equity-efficiency interactions in cost-effectiveness analysis-three approaches applied to breast cancer control. Value Health 13(5), 573–579 (2010)CrossRefPubMed
103.
Agapova, M., et al.: Applying quantitative benefit-risk analysis to aid regulatory decision making in diagnostic imaging: methods, challenges, and opportunities. Acad Radiol 21(9), 1138 (2014)CrossRefPubMed
104.
Tervonen, T., et al.: MCDA swing weighting and discrete choice experiments for elicitation of patient benefit-risk preferences: a critical assessment. Pharmacoepidemiol Drug Saf 26(12), 1483–1491 (2017)CrossRefPubMed
105.
106.
Peacock, S., et al.: Overcoming barriers to priority setting using interdisciplinary methods. Health Policy 92(2–3), 124–132 (2009)CrossRefPubMed
107.
Craig, L.E., et al.: Approaches to economic evaluations of stroke rehabilitation. Int J Stroke 9(1), 88–100 (2014)CrossRefPubMed
108.
Carrera, P., Ijzerman, M.J.: Are current ICER thresholds outdated? Valuing medicines in the era of personalized healthcare. Expert Rev Pharmacoecon Outcomes Res 16(4), 435–437 (2016)CrossRefPubMed
109.
Paolucci, F., et al.: Decision making and priority setting: the evolving path towards universal health coverage. Appl Health Econ Health Policy 15(6), 697–706 (2017)CrossRefPubMed
110.
111.
Simoens, S.: Health technologies for rare diseases: does conventional HTA still apply? Expert Rev Pharmacoecon Outcomes Res 14(3), 315 (2014)CrossRefPubMed
112.
113.
Rotter, J.S., Foerster, D., Bridges, J.F.: The changing role of economic evaluation in valuing medical technologies. Expert Rev Pharmacoecon Outcomes Res 12(6), 711–723 (2012)CrossRefPubMed
114.
Mühlbacher, A.C., Johnson, F.R.: Giving patients a meaningful voice in European health technology assessments: the role of health preference research. Patient: Patient Cent Outcomes Res 10(4), 527–530 (2017)CrossRef
115.
Xie, F., et al.: Using health technology assessment to support evidence-based decision-making in Canada: an academic perspective. Expert Rev Pharmacoecon Outcomes Res 11(5), 513–521 (2011)CrossRefPubMed
116.
Diaby, V., et al.: Multi-criteria decision analysis for health technology assessment in Canada: insights from an expert panel discussion. Expert Rev Pharmacoecon Outcomes Res 15(1), 13 (2015)CrossRefPubMed
117.
Kolasa, K., Kalo, Z., Zah, V.: The use of non-economic criteria in pricing and reimbursement decisions in Central and Eastern Europe: issues, trends and recommendations. Expert Rev Pharmacoecon Outcomes Res 16(4), 483 (2016)CrossRefPubMed
118.
Angelis, A., Kanavos, P., Montibeller, G.: Resource allocation and priority setting in health care: a multi-criteria decision analysis problem of value? Glob Policy 8, 76–83 (2017)CrossRef
119.
Towse, A.: Net clinical benefit: the art and science of jointly estimating benefits and risks of medical treatment. Value Health 13(Suppl 1), S30–S32 (2010)CrossRefPubMed
120.
Sculpher, M., Claxton, K., Pearson, S.D.: Developing a value framework: the need to reflect the opportunity costs of funding decisions. Value Health 20, 234–239 (2017)CrossRefPubMed
121.
Marsh, K., et al.: The use of MCDA in HTA: great potential, but more effort needed. Value Health 21(4), 394–397 (2017)CrossRefPubMed
122.
Garattini, L., Padula, A.: Multiple criteria decision analysis in health technology assessment for drugs: just another illusion? Appl Health Econ Health Policy 16(1), 1–4 (2017)CrossRef
123.
Baltussen, R., et al.: Multi-criteria decision analysis to prioritize health interventions: capitalizing on first experiences. Health Policy 96(3), 262–264 (2010)CrossRefPubMed
124.
Rosselli, D., et al.: HTA implementation in Latin American countries: comparison of current and preferred status. Value Health Reg Issues 14, 20–27 (2017)CrossRefPubMed
125.
Endrei, D., Molics, B., Ágoston, I.: Multicriteria decision analysis in the reimbursement of new medical technologies: real-world experiences from Hungary. Value Health 17(4), 487–489 (2014)CrossRefPubMed
126.
Levitan, B., Phillips, L.D., Walker, S.: Structured approaches to benefit-risk assessment: a case study and the patient perspective. Ther Innov Regul Sci 48(5), 564–573 (2014)PubMed
127.
Radaelli, G., et al.: Implementation of EUnetHTA core Model(R) in Lombardia: the VTS framework. Int J Technol Assess Health Care 30(1), 105–112 (2014)CrossRefPubMed
128.
Liberti, L., McAuslane, J.N., Walker, S.: Standardizing the benefit-risk assessment of new medicines. Pharm Med N Z 25(3), 139–146 (2011)CrossRef
129.
Dionne, F., et al.: Developing a multi-criteria approach for drug reimbursement decision making: an initial step forward. J Popul Ther Clin Pharmacol 22(1), e68 (2015)PubMed
130.
Hallgreen, C.E., et al.: Benefit-risk assessment in a post-market setting: a case study integrating real-life experience into benefit-risk methodology. Pharmacoepidemiol Drug Saf 23(9), 974–983 (2014)CrossRefPubMed
131.
Sarac, S.B., et al.: A comprehensive approach to benefit-risk assessment in drug development. Basic Clin Pharmacol Toxicol 111(1), 65–72 (2012)PubMed
132.
Migliore, A., et al.: Is it the time to rethink clinical decision-making strategies? From a single clinical outcome evaluation to a clinical multi-criteria decision assessment (CMDA). Med Hypotheses 85, 433–440 (2015)CrossRefPubMed
133.
Agapova, M., et al.: A proposed approach for quantitative benefit-risk assessment in diagnostic radiology guideline development: the American College of Radiology Appropriateness Criteria Example. J Eval Clin Pract 23(1), 128–138 (2017)CrossRefPubMed
134.
Tsiachristas, A., et al.: Broader economic evaluation of disease management programs using multi-criteria decision analysis. Int J Technol Assess Health Care 29(3), 301–308 (2013)CrossRefPubMed
135.
Linkov, I., Satterstrom, F.K., Corey, L.M.: Nanotoxicology and nanomedicine: making hard decisions. Nanomed Nanotechnol Biol Med 4(2), 167 (2008)CrossRef
136.
Diaby, V., Laurier, C., Lachaine, J.: A Proposed framework for formulary listing in low-income countries. Pharm Med N Z 25(2), 71–82 (2011)CrossRef
137.
138.
Danko, D., Molnar, M.P.: Balanced assessment systems revisited. J Mark Acess Health Policy 5(1), 1355190 (2017)CrossRef
139.
Husereau, D., Boucher, M., Noorani, H.: Priority setting for health technology assessment at CADTH. Int J Technol Assess Health Care 26(3), 341 (2010)CrossRefPubMed
140.
Poulin, P., et al.: Multi-criteria development and incorporation into decision tools for health technology adoption. J Health Organ Manag 2, 246 (2013)CrossRef
141.
Goetghebeur, M.M., et al.: Evidence and Value: impact on DEcisionMaking–the EVIDEM framework and potential applications. BMC Health Serv Res 8, 270 (2008)CrossRefPubMedPubMedCentral
142.
Wagner, M., et al.: Can the EVIDEM Framework tackle issues raised by evaluating treatments for rare diseases: analysis of issues and policies, and context-specific adaptation. Pharmacoeconomics 34(3), 285–301 (2016)CrossRefPubMed
143.
Golan, O., Hansen, P.: Which health technologies should be funded? A prioritization framework based explicitly on value for money. Isr J Health Policy Res 1(1), 44 (2012)CrossRefPubMedPubMedCentral
144.
Agapova, M., et al.: Toward a framework for benefit-risk assessment in diagnostic imaging: identifying scenario-specific criteria. Acad Radiol 24(5), 538–549 (2017)CrossRefPubMed
145.
146.
Timmis, J.K., Rigat, F., Rappuoli, R.: Core values for vaccine evaluation. Vaccine 35(Suppl 1), A57–A62 (2017)CrossRefPubMed
147.
Kaslow, D.C., et al.: The role of vaccines and vaccine decision-making to achieve the goals of the Grand Convergence in public health. Vaccine 35(Suppl 1), A10–A15 (2017)CrossRefPubMed
148.
Brixner, D., et al.: Considering multicriteria decision analysis (MCDA) simple scoring as an evidence-based HTA methodology for evaluating off-patent pharmaceuticals (OPPs) in emerging markets. Value Health Reg Issues 13, 1–6 (2017)CrossRefPubMed
149.
150.
Vogler, S., Zimmermann, N., Habimana, K.: Stakeholder preferences about policy objectives and measures of pharmaceutical pricing and reimbursement. Health Policy Technol 5, 213–225 (2016)CrossRef
151.
Zelei, T., et al.: Systematic review on the evaluation criteria of orphan medicines in Central and Eastern European countries. Orphanet J Rare Dis 11(1), 72 (2016)CrossRefPubMedPubMedCentral
152.
Polisena, J., et al.: Case studies that illustrate disinvestment and resource allocation decision-making processes in health care: a systematic review. Int J Technol Assess Health Care 29(2), 174 (2013)CrossRefPubMed
153.
Wernz, C., Zhang, H., Phusavat, K.: International study of technology investment decisions at hospitals. Ind Manag Data Syst 114(4), 568–582 (2014)CrossRef
154.
Gurtner, S.: Making the right decisions about new technologies: a perspective on criteria and preferences in hospitals. Health Care Manag Rev 39(3), 245–254 (2014)CrossRef
155.
Antoñanzas, F., Terkola, R., Postma, M.: The value of medicines: a crucial but vague concept. PharmacoEconomics 34(12), 1227–1239 (2016)CrossRefPubMed
156.
Jehu-Appiah, C., et al.: Balancing equity and efficiency in health priorities in Ghana: the use of multicriteria decision analysis. Value Health 11, 1081–1087 (2008)CrossRefPubMed
157.
158.
Schmitz, S., et al.: Identifying and revealing the importance of decision-making criteria for health technology assessment: a retrospective analysis of reimbursement recommendations in ireland. PharmacoEconomics 34(9), 925–937 (2016)CrossRefPubMed
159.
Johri, M., Norheim, O.F.: Can cost-effectiveness analysis integrate concerns for equity? Systematic review. Int J Technol Assess Health Care 28(2), 125–132 (2012)CrossRefPubMed
160.
Dionne, F., et al.: The challenge of obtaining information necessary for multi-criteria decision analysis implementation: the case of physiotherapy services in Canada. Cost Eff Resour Alloc 11(1), 11–26 (2013)CrossRefPubMedPubMedCentral
161.
Chen, T.Y.: A signed-distance-based approach to importance assessment and multi-criteria group decision analysis based on interval type-2 fuzzy set. Knowl Inf Syst 35(1), 193–231 (2013)CrossRef
162.
Broekhuizen, H., et al.: Estimating the value of medical treatments to patients using probabilistic multi criteria decision analysis. BMC Med Inform Decis Mak 15(1), 102 (2015)CrossRefPubMedPubMedCentral
163.
Broekhuizen, H., et al.: Weighing clinical evidence using patient preferences: an application of probabilistic multi-criteria decision analysis. Pharmacoeconomics 35(3), 259–269 (2017)CrossRefPubMed
164.
Wen, S., Zhang, L., Yang, B.: Two approaches to incorporate clinical data uncertainty into multiple criteria decision analysis for benefit-risk assessment of medicinal products. Value Health 17(5), 619–628 (2014)CrossRefPubMed
165.
van Valkenhoef, G., et al.: Multicriteria benefit-risk assessment using network meta-analysis. J Clin Epidemiol 65(4), 394–403 (2012)CrossRefPubMed
166.
Wang, Y., Mai, Y., He, W.: A quantitative approach for benefit-risk assessment using stochastic multi-criteria discriminatory method. Stat Biopharm Res 8(4), 373–378 (2016)CrossRef
167.
Postmus, D., et al.: A multi-criteria decision analysis perspective on the health economic evaluation of medical interventions. Eur J Oper Res 15(7), 709–716 (2014)
168.
Hutter, M., Rodríguez-Ibeas, R., Antonanzas, F.: Methodological reviews of economic evaluations in health care: what do they target? Eur J Health Econ 15(8), 829–840 (2014)CrossRefPubMed
169.
Belton, V., Gear, T.: On a shortcoming of Saaty’s method of analytic hierarchies. Omega 11(3), 228–230 (1983)CrossRef
170.
Dyer, J.S.: Remarks on the analytic hierarchy process. Manag Sci 36(3), 249–258 (1990)CrossRef
171.
Bana-e-Costa, C.A., Vansnick, J.C.: A critical analysis of the eigenvalue method used to derive priorities in AHP. Eur J Oper Res 187(3), 1422–1428 (2008)CrossRef
172.
Bana-e-Costa, C.A., Oliveira, M.D.: A multicriteria decision analysis model for faculty evaluation. Omega 40, 424–436 (2012)CrossRef
173.
Stewart, T.J., Durbach, I.: Dealing with uncertainties in MCDA. In: Greco, S., Ehrgott, M., Figueira, J. (eds.) Multiple Criteria Decision Analysis, pp. 467–496. International Series in Operations Research and Management Science, New York (2016)CrossRef
174.
Bana-e-Costa, C.A., et al.: Using the MACBETH socio-technical approach to build the EURO-HEALTHY PHI. In: Santana, P. (ed.) Promoting Population Health and Equity in Europe: From Evidence to Policy, pp. 71–77. University of Coimbra, Coimbra (2017)
175.
Santana, P., et al.: Atlas of Population Health in European Union Regions. Imprensa da Universidade de Coimbra, Coimbra (2017)CrossRef
176.
Gooyert, V.D., et al.: Reviewing the role of stakeholders in operational research: a stakeholder theory perspective. Eur J Oper Res 262(2), 402–410 (2017)CrossRef
177.
de Vaus, D.: Surveys in Social Research, 6th edn. Routledge, London (2014)
178.
Lewis, L.F.: Group support systems: overview and guided tour. In: Kilgour, D.M., Eden, C. (eds.) Handbook of Group Decision and Negotiation, pp. 249–268. Springer, Netherlands (2010)CrossRef
179.
Lienert, J., Duygan, M., Zheng, J.: Preference stability over time with multiple elicitation methods to support wastewater infrastructure decision-making. Eur J Oper Res 253(3), 746–760 (2016)CrossRef
180.
Bana-e-Costa, C.A., Vansnick, J.-C.: MACBETH—an interactive path towards the construction of cardinal value functions. Int Trans Oper Res 1(4), 489–500 (1994)CrossRef
181.
Bana-e-Costa, C.A., De Corte, J.-M., Vansnick, J.-C.: MACBETH. Int J Inform Technol Decis Mak 11(2), 359–387 (2012)CrossRef
182.
183.
Morton, A., Fasolo, B.: Behavioural decision theory for multi-criteria decision analysis: a guided tour. J Oper Res Soc 60, 268–275 (2009)CrossRef
184.
Shephard, G.G., Kirkwood, C.W.: Managing the judgmental probability elicitation process: a case study of analyst/manager interaction. IEEE Trans Eng Manag 41(4), 414–425 (1994)CrossRef
185.
Schein, E.H.: Process consultation revisited: building the helping relationship. Addison-Wesley, Reading (1999)
186.
Schuman, S. (ed.): The IAF handbook of group facilitation: best practices from the leading organization in facilitation. The Jossey-Bass Business and Management Series, p. 664. Jossey-Bass, San Francisco (2005)
187.
Phillips, L.D.: Decision conferencing. In: Edwards, W., Miles, R.F., von Winterfeldt, D. (eds.) Advances in Decision Analysis: From Foundations to Applications, pp. 375–399. Cambridge University Press, Cambridge (2007)CrossRef
188.
Langhans, S.D., Lienert, J.: Four Common Simplifications of Multi-Criteria Decision Analysis do not hold for River Rehabilitation. PLoS ONE 11(3), e0150695 (2016)CrossRefPubMedPubMedCentral
189.
IST: Deliverable 6.2 of WP6: Development of the multicriteria model to evaluate Population Health. Instituto Superior Técnico, Lisbon (2017)
190.
Oliveira, M.D., et al.: Prioritizing health care interventions: a multicriteria resource allocation model to inform the choice of community care programmes. In: Tànfani, E., Testi, A. (eds.) Advanced decision making methods applied to health care, pp. 141–154. Springer, Milan (2012)CrossRef
191.
Oliveira, M.D., Bevan, G.: Modelling the redistribution of hospital supply to achieve equity taking account of patient’s behaviour. Health Care Manag Sci 9(1), 19–30 (2006)CrossRefPubMed
192.
Mestre, A., Oliveira, M.D., Barbosa-Póvoa, A.: Location-allocation approaches for hospital network planning under uncertainty. Eur J Oper Res 240, 791–806 (2015)CrossRef
193.
Cardoso, T., et al.: Moving towards an equitable long-term care network: a multi-objective and multi-period planning approach. Omega 58, 69–85 (2016)CrossRef
194.
Marttunen, M., Lienert, J., Belton, V.: Structuring problems for multi-criteria decision analysis in practice: a literature review of method combinations. Eur J Oper Res 263(1), 1–17 (2017)CrossRef
195.
Rodrigues, T.R., et al.: Modelling multicriteria value interactions with reasoning maps. Eur J Oper Res 3(1), 1054–1071 (2017)CrossRef
196.
Bana-e-Costa, C.A., Carnero, M.C., Oliveira, M.D.: A multi-criteria model for auditing a predictive maintenance programme. Eur J Oper Res 217(2), 381–393 (2012)CrossRef
197.
Raiffa, H.: Decision Analysis: Introductory Lectures on Choices Under Uncertainty. Random House, New York (1968)
198.
Oliveira, M.D., Lopes, D.F., Bana-e-Costa, C.A.: Improving occupational health and safety risk evaluation through decision analysis. Int Trans Oper Res 25(1), 375–403 (2018)CrossRef
199.
Kilgour, D.M., Eden, C.: Handbook of Group Decision and Negotiation: Advances in Group Decision and Negotiation. Springer, Netherlands (2010)CrossRef
200.
201.
Bana-e-Costa, C.A.: Readings in Multiple Criteria Decision Aid. Springer, Heidelberg (1990)CrossRef
202.
Postmus, D., et al.: A multi-criteria decision analysis perspective on the health economic evaluation of medical interventions. Eur J Health Econ 15, 709–716 (2014)CrossRefPubMed
203.
French, S., Maule, J., Papamichail, N.: Decision Behaviour, Analysis and Support. Cambridge University Press, Cambridge (2009)CrossRef
Metadata
Title
Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art
Authors
Mónica D. Oliveira
Inês Mataloto
Panos Kanavos
Publication date
01-08-2019
Publisher
Springer Berlin Heidelberg
Published in
The European Journal of Health Economics / Issue 6/2019
Print ISSN: 1618-7598
Electronic ISSN: 1618-7601
DOI
https://doi.org/10.1007/s10198-019-01052-3

Other articles of this Issue 6/2019

The European Journal of Health Economics 6/2019 Go to the issue

Original Paper

Accuracy of budget impact estimations and impact on patient access: a hepatitis C case study

Original Paper

Moral hazard under zero price policy: evidence from Japanese long-term care claims data

Original Paper

How to make value-based health insurance designs more effective? A systematic review and meta-analysis

Original Paper

Health state utilities associated with post-surgical Staphylococcus aureus infections

Original Paper

Health-related quality of life measured with the EQ-5D-5L: estimation of normative index values based on a representative German population sample and value set

Editorial

Assessing capability in economic evaluation: a life course approach?