Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Cost Effectiveness and Resource Allocation
Published in: Cost Effectiveness and Resource Allocation 1/2014

Open Access 01-12-2014 | Research

Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework

Authors: Janine van Til, Catharina Groothuis-Oudshoorn, Marijke Lieferink, James Dolan, Mireille Goetghebeur

Published in: Cost Effectiveness and Resource Allocation | Issue 1/2014

Login to get access

Abstract

Background

There is an increased interest in the use of multi-criteria decision analysis (MCDA) to support regulatory and reimbursement decision making. The EVIDEM framework was developed to provide pragmatic multi-criteria decision support in health care, to estimate the value of healthcare interventions, and to aid in priority-setting. The objectives of this study were to test 1) the influence of different weighting techniques on the overall outcome of an MCDA exercise, 2) the discriminative power in weighting different criteria of such techniques, and 3) whether different techniques result in similar weights in weighting the criteria set proposed by the EVIDEM framework.

Methods

A sample of 60 Dutch and Canadian students participated in the study. Each student used an online survey to provide weights for 14 criteria with two different techniques: a five-point rating scale and one of the following techniques selected randomly: ranking, point allocation, pairwise comparison and best worst scaling.

Results

The results of this study indicate that there is no effect of differences in weights on value estimates at the group level. On an individual level, considerable differences in criteria weights and rank order occur as a result of the weight elicitation method used, and the ability of different techniques to discriminate in criteria importance. Of the five techniques tested, the pair-wise comparison of criteria has the highest ability to discriminate in weights when fourteen criteria are compared.

Conclusions

When weights are intended to support group decisions, the choice of elicitation technique has negligible impact on criteria weights and the overall value of an innovation. However, when weights are used to support individual decisions, the choice of elicitation technique influences outcome and studies that use dissimilar techniques cannot be easily compared. Weight elicitation through pairwise comparison of criteria is preferred when taking into account its superior ability to discriminate between criteria and respondents’ preferences.
Appendix
Available only for authorised users
Literature
1.
Guindo LA, Wagner M, Baltussen R, Rindress D, van Til J, Kind P, Goetghebeur MM: From efficacy to equity: literature review of decision criteria for resource allocation and healthcare decision making. Cost Eff Resour Alloc 2012,10(1):9. 10.1186/1478-7547-10-9PubMedCentralPubMedCrossRef
2.
Baltussen R, Niessen L: Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc 2006, 4: 14. 10.1186/1478-7547-4-14PubMedCentralPubMedCrossRef
3.
Tanimoto T, Tsubokura M, Mori J, Pietrek M, Ono S, Kami M: Differences in drug approval processes of 3 regulatory agencies: a case study of gemtuzumab ozogamicin. Invest New Drugs 2013,31(2):473–478. 10.1007/s10637-012-9877-8PubMedCrossRef
4.
Thokala P, Duenas A: Multiple criteria decision analysis for health technology assessment. Value Health 2012,15(8):1172–1181. 10.1016/j.jval.2012.06.015PubMedCrossRef
5.
Belton VSTJ: Multiple Criteria Decision Analysis: an Integrated Approach. Boston: Kluwer Academic Publishers; 2002.CrossRef
6.
Mt-Isa S, Hallgreen CE, Wang N, Callreus T, Genov G, Hirsch I, Hobbiger SF, Hockley KS, Luciani D, Phillips LD, Quartey G, Sarac SB, Stoeckert I, Tzoulaki I, Micaleff A, Ashby D: Balancing benefit and risk of medicines: a systematic review and classification of available methodologies. Pharmacoepidemiol Drug Saf 2014,23(7):667–678. 10.1002/pds.3636PubMedCrossRef
7.
Devlin NJ, Sussex J: Incorporating Multiple Criteria in HTA: Methods and Processes. London: Office of Health Economics; 2011.
8.
Diaby V, Campbell K, Goeree R: Multi-criteria decision analysis (MCDA) in health care: a bibliometric analysis. Oper Res Health Care 2013,2(1–2):20–24.CrossRef
9.
Tony M, Wagner M, Khoury H, Rindress D, Papastavros T, Oh P, Goetghebeur MM: Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada. BMC Health Serv Res 2011, 11: 329. 10.1186/1472-6963-11-329PubMedCentralPubMedCrossRef
10.
Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D: Evidence and value: impact on DEcisionMaking–the EVIDEM framework and potential applications. BMC Health Serv Res 2008, 8: 270. 10.1186/1472-6963-8-270PubMedCentralPubMedCrossRef
11.
Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D: Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): applying the EVIDEM framework to medicines appraisal. Med Decis Mak 2012,32(2):376–388. 10.1177/0272989X11416870CrossRef
12.
Goetghebeur MM, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C: Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decision-making framework to growth hormone for Turner syndrome patients. Cost Eff Resour Alloc 2010, 8: 4. 10.1186/1478-7547-8-4PubMedCentralPubMedCrossRef
13.
Miot J, Wagner M, Khoury H, Rindress D, Goetghebeur MM: Field testing of a multicriteria decision analysis (MCDA) framework for coverage of a screening test for cervical cancer in South Africa. Cost Eff Resour Alloc 2012,10(1):2. 10.1186/1478-7547-10-2PubMedCentralPubMedCrossRef
14.
Radaelli G, Lettieri E, Masella C, Merlino L, Strada A, Tringali M: Implementation of EUnetHTA core Model(R) in Lombardia: the VTS framework. Int J Technol Assess Health Care 2014,30(1):105–112. 10.1017/S0266462313000639PubMedCrossRef
15.
Liberatore MJ, Nydick RL: The analytic hierarchy process in medical and health care decision making: a literature review. Eur J Oper Res 2008,189(1):194–207. 10.1016/j.ejor.2007.05.001CrossRef
16.
Edward W, Barron FH: SMARTS and SMARTER: Improved Simple Methods for Multiattribute Utility Measurement. 60th edition. Kidlington: ROYAUME-UNI: Elsevier; 1994.
17.
Dolan JG: Multi-criteria clinical decision support: a primer on the use of multiple criteria decision making methods to promote evidence-based, patient-centered healthcare. Patient 2010,3(4):229–248. 10.2165/11539470-000000000-00000PubMedCentralPubMedCrossRef
18.
Weber M, Borcherding K: Behavioral influences on weight judgments in multiattribute decision making. Eur J Oper Res 1993,67(1):1–12. 10.1016/0377-2217(93)90318-HCrossRef
19.
Edwards W, Fasolo B: Decision technology. Annu Rev Psychol 2001, 52: 581–606. 10.1146/annurev.psych.52.1.581PubMedCrossRef
20.
Doyle JR, Green RH, Bottomley PA: Judging relative importance: direct rating and point allocation are not equivalent. Organ Behav Hum Decis Process 1997,70(1):65–72. 10.1006/obhd.1997.2694PubMedCrossRef
21.
Dolan JG, Isselhardt BJ Jr, Cappuccio JD: The analytic hierarchy process in medical decision making: a tutorial. Med Decis Mak 1989,9(1):40–50. 10.1177/0272989X8900900108CrossRef
22.
Flynn TN, Louviere JJ, Peters TJ, Coast J: Best-worst scaling: what it can do for health care research and how to do it. J Health Econ 2007,26(1):171–189. 10.1016/j.jhealeco.2006.04.002PubMedCrossRef
23.
Kuhfeld WF, Tobias RD, Garratt M: Efficient experimental design with marketing research applications. J Mark Res 1995, 31: 545–557.CrossRef
24.
Nickerson CA: Assessing convergent validity of health-state utilities obtained using different scaling methods. Med Decis Making 1999,19(4):487–498. 10.1177/0272989X9901900417PubMedCrossRef
25.
Martin Bland J, Altman D: Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986,327(8476):307–310. 10.1016/S0140-6736(86)90837-8CrossRef
26.
Altman DG, Bland JM: Measurement in medicine: the analysis of method comparison studies. The Statistician 1983,32(3):307–317. 10.2307/2987937CrossRef
27.
van Til JA, Dolan JG, Stiggelbout AM, Groothuis KC, Ijzerman MJ: The use of multi-criteria decision analysis weight elicitation techniques in patients with mild cognitive impairment: a pilot study. Patient 2008,1(2):127–135. 10.2165/01312067-200801020-00008PubMedCrossRef
28.
Sato Y: How to measure human perception in survey questionnaires. Int J Anal Hierarchy Process 2009,1(2):64–82.
29.
Youngkong S, Tromp N, Chitama D: The EVIDEM framework and its usefulness for priority setting across a broad range of health interventions. Cost Eff Resour Alloc 2011, 9: 16. 10.1186/1478-7547-9-16PubMedCentralPubMedCrossRef
30.
Tanios N, Wagner M, Tony M, Baltussen R, van Til J, Rindress D, Kind P, Goetghebeur MM: Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers. Int J Technol Assess Health Care 2013,29(4):456–465. 10.1017/S0266462313000573PubMedCrossRef
31.
Scholz SW, Meissner M, Decker R: Measuring consumer preferences for complex products: a compositional approach based on paired comparisons. J Mark Res 2010,47(4):685–698. 10.1509/jmkr.47.4.685CrossRef
32.
Lichtenstein S, Slovic P: The Construction of Preference. Cambridge, England: Cambridge University Press; 2006.CrossRef
Metadata
Title
Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework
Authors
Janine van Til
Catharina Groothuis-Oudshoorn
Marijke Lieferink
James Dolan
Mireille Goetghebeur
Publication date
01-12-2014
Publisher
BioMed Central
Published in
Cost Effectiveness and Resource Allocation / Issue 1/2014
Electronic ISSN: 1478-7547
DOI
https://doi.org/10.1186/1478-7547-12-22

Other articles of this Issue 1/2014

Cost Effectiveness and Resource Allocation 1/2014 Go to the issue

Research

Estimating the cost-effectiveness of nutrition supplementation for malnourished, HIV-infected adults starting antiretroviral therapy in a resource-constrained setting

Research

Comparing the cost effectiveness of harm reduction strategies: a case study of the Ukraine

Research

Budget constraint and vaccine dosing: a mathematical modelling exercise

Research

Cost-effectiveness of community vegetable gardens for people living with HIV in Zimbabwe

Research

Ownership and technical efficiency of hospitals: evidence from Ghana using data envelopment analysis

Research

Economic evaluation of policy initiatives in the organisation and delivery of healthcare: a case study of gastroenterology endoscopy services