Skip to main content
Top
Published in: BMC Medical Research Methodology 1/2020

Open Access 01-12-2020 | Research article

Which interactions matter in economic evaluations? A systematic review and simulation study

Authors: Helen Dakin, Alastair Gray

Published in: BMC Medical Research Methodology | Issue 1/2020

Login to get access

Abstract

Background

We aimed to assess the magnitude of interactions in costs, quality-adjusted life-years (QALYs) and net benefits within a sample of published economic evaluations of factorial randomised controlled trials (RCTs), evaluate the impact that different analytical methods would have had on the results and compare the performance of different criteria for identifying which interactions should be taken into account.

Methods

We conducted a systematic review of full economic evaluations conducted alongside factorial RCTs and reviewed the methods used in different studies, as well as the incidence, magnitude, statistical significance, and type of interactions observed within the trials. We developed the interaction-effect ratio as a measure of the magnitude of interactions relative to main effects. For those studies reporting sufficient data, we assessed whether changing the form of analysis to ignore or include interactions would have changed the conclusions. We evaluated how well different criteria for identifying which interactions should be taken into account in the analysis would perform in practice, using simulated data generated to match the summary statistics of the studies identified in the review.

Results

Large interactions for economic endpoints occurred frequently within the 40 studies identified in the review, although interactions rarely changed the conclusions.

Conclusions

Simulation work demonstrated that in analyses of factorial RCTs, taking account of all interactions or including interactions above a certain size (regardless of statistical significance) minimised the opportunity cost from adopting treatments that do not in fact have the highest true net benefit.
Appendix
Available only for authorised users
Footnotes
1
The interaction:effect ratio differs from the “interaction ratio” used by McAlister et al. [10]. McAlister’s interaction ratio is simply the relative effect (e.g. odds ratio) of A vs. not-A for patients also receiving B, divided by the relative effect of A vs. not-A for patients not receiving B (interaction ratio = (oddsab/oddsb) ÷ (oddsa/odds0)) and therefore equals the interaction on a logarithmic scale (interaction on log-scale = exp[(ln[oddsab] − ln[oddsb]) − (ln[oddsa] − ln[odds0])]). Unlike our interaction:effect ratio, McAlister’s interaction ratio is appropriate only for data interpreted on a multiplicative scale and does not distinguish between qualitative and non-qualitative interactions. At least one previous paper has used the interaction divided by simple effect to describe the ranges of interaction magnitude in which different analytical approaches performed best [3]. However, this study did not include any adjustment for mixed interactions, did not link the ranges of ratio values with different types of interaction and did not propose this ratio as a method for describing interactions in general.
 
2
One study meeting inclusion criteria was terminated early due to poor recruitment but was published as a monograph without analysis of economic results; this is considered in the review alongside protocols.
 
3
Since four of these studies were larger than 2 × 2 or reported results by subgroup, this gave 24 interactions for each of three outcomes (cost, QALYs and NMB): 72 interactions in total.
 
4
Three of these studies were included in the review as the authors described them as factorial [3840]. In one study [40], general practitioners randomised to one of the three active treatment groups received a training session not given to the control group. Conversely, two trials gave patients in the control group an additional intervention not given to the other three groups. A fourth trial, in which the second factor compared physiotherapy against reinforcement of the advice given as part of factor 1 (a whiplash book or usual advice), was excluded from the review as the authors did not describe it as factorial [41]. A further study that was included in the review allowed information sharing between practitioners within the ab group that was not possible within the groups receiving < 2 interventions [19].
 
Literature
1.
go back to reference Dakin HA, Gray A. Economic evaluation of factorial randomised controlled trials: challenges, methods and recommendations. Stat Med. 2017;36(18):2814–30.CrossRef Dakin HA, Gray A. Economic evaluation of factorial randomised controlled trials: challenges, methods and recommendations. Stat Med. 2017;36(18):2814–30.CrossRef
2.
go back to reference Dakin HA, Gray A. Decision-making for healthcare resource allocation: joint versus separate decisions on interacting interventions. Med Decis Mak. 2018;38(4):476–86.CrossRef Dakin HA, Gray A. Decision-making for healthcare resource allocation: joint versus separate decisions on interacting interventions. Med Decis Mak. 2018;38(4):476–86.CrossRef
3.
go back to reference Hung HM, Chi GY, O'Neill RT. Efficacy evaluation for monotherapies in two-by-two factorial trials. Biometrics. 1995;51(4):1483–93.CrossRef Hung HM, Chi GY, O'Neill RT. Efficacy evaluation for monotherapies in two-by-two factorial trials. Biometrics. 1995;51(4):1483–93.CrossRef
4.
go back to reference Hung HM. Two-stage tests for studying monotherapy and combination therapy in two-by-two factorial trials. Stat Med. 1993;12(7):645–60.CrossRef Hung HM. Two-stage tests for studying monotherapy and combination therapy in two-by-two factorial trials. Stat Med. 1993;12(7):645–60.CrossRef
5.
go back to reference Montgomery AA, Peters TJ, Little P. Design, analysis and presentation of factorial randomised controlled trials. BMC Med Res Methodol. 2003;3:26.CrossRef Montgomery AA, Peters TJ, Little P. Design, analysis and presentation of factorial randomised controlled trials. BMC Med Res Methodol. 2003;3:26.CrossRef
6.
go back to reference Brittain E, Wittes J. Factorial designs in clinical trials: the effects of non-compliance and subadditivity. Stat Med. 1989;8(2):161–71.CrossRef Brittain E, Wittes J. Factorial designs in clinical trials: the effects of non-compliance and subadditivity. Stat Med. 1989;8(2):161–71.CrossRef
7.
go back to reference Green S. Factorial designs with time to event endpoints. In: Crowley J, Ankerst DP, editors. Handbook of Statistics in Clinical Oncology. 2nd ed. Boca Raton: Chapman & Hall/CRC; 2006. p. 181–9. Green S. Factorial designs with time to event endpoints. In: Crowley J, Ankerst DP, editors. Handbook of Statistics in Clinical Oncology. 2nd ed. Boca Raton: Chapman & Hall/CRC; 2006. p. 181–9.
8.
go back to reference Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. J Health Econ. 1999;18(3):341–64.CrossRef Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. J Health Econ. 1999;18(3):341–64.CrossRef
9.
go back to reference Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.CrossRef Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.CrossRef
10.
go back to reference McAlister FA, Straus SE, Sackett DL, Altman DG. Analysis and reporting of factorial trials: a systematic review. JAMA. 2003;289(19):2545–53.CrossRef McAlister FA, Straus SE, Sackett DL, Altman DG. Analysis and reporting of factorial trials: a systematic review. JAMA. 2003;289(19):2545–53.CrossRef
11.
go back to reference Brown J, Welton NJ, Bankhead C, Richards SH, Roberts L, Tydeman C, Peters TJ. A Bayesian approach to analysing the cost-effectiveness of two primary care interventions aimed at improving attendance for breast screening. Health Econ. 2006;15(5):435–45.CrossRef Brown J, Welton NJ, Bankhead C, Richards SH, Roberts L, Tydeman C, Peters TJ. A Bayesian approach to analysing the cost-effectiveness of two primary care interventions aimed at improving attendance for breast screening. Health Econ. 2006;15(5):435–45.CrossRef
12.
go back to reference Welton NJ, Ades AE, Caldwell DM, Peters TJ. Research prioritization based on expected value of partial perfect information: a case-study on interventions to increase uptake of breast cancer screening. J R Statist Soc A. 2008;171(Part 4):807–41.CrossRef Welton NJ, Ades AE, Caldwell DM, Peters TJ. Research prioritization based on expected value of partial perfect information: a case-study on interventions to increase uptake of breast cancer screening. J R Statist Soc A. 2008;171(Part 4):807–41.CrossRef
13.
go back to reference Welton NJ, Madan JJ, Caldwell DM, Peters TJ, Ades AE. Expected value of sample information for multi-arm cluster randomized trials with binary outcomes. Med Decis Mak. 2013;34:352–65.CrossRef Welton NJ, Madan JJ, Caldwell DM, Peters TJ, Ades AE. Expected value of sample information for multi-arm cluster randomized trials with binary outcomes. Med Decis Mak. 2013;34:352–65.CrossRef
15.
go back to reference UK Beam Trial Team. United Kingdom back pain exercise and manipulation (UK BEAM) randomised trial: cost effectiveness of physical treatments for back pain in primary care. BMJ. 2004;329(7479):1381. UK Beam Trial Team. United Kingdom back pain exercise and manipulation (UK BEAM) randomised trial: cost effectiveness of physical treatments for back pain in primary care. BMJ. 2004;329(7479):1381.
16.
go back to reference Boyle J, McCartney E, Forbes J, O'Hare A: A randomised controlled trial and economic evaluation of direct versus indirect and individual versus group modes of speech and language therapy for children with primary language impairment. Health Technol Assess. 2007;11(25):iii-iv, xi-xii, 1–139. Boyle J, McCartney E, Forbes J, O'Hare A: A randomised controlled trial and economic evaluation of direct versus indirect and individual versus group modes of speech and language therapy for children with primary language impairment. Health Technol Assess. 2007;11(25):iii-iv, xi-xii, 1–139.
17.
go back to reference Hollinghurst S, Sharp D, Ballard K, Barnett J, Beattie A, Evans M, Lewith G, Middleton K, Oxford F, Webley F, et al. Randomised controlled trial of Alexander technique lessons, exercise, and massage (ATEAM) for chronic and recurrent back pain: economic evaluation. BMJ. 2008;337:a2656.CrossRef Hollinghurst S, Sharp D, Ballard K, Barnett J, Beattie A, Evans M, Lewith G, Middleton K, Oxford F, Webley F, et al. Randomised controlled trial of Alexander technique lessons, exercise, and massage (ATEAM) for chronic and recurrent back pain: economic evaluation. BMJ. 2008;337:a2656.CrossRef
18.
go back to reference Reddy P, Kalus JS, Caron MF, Horowitz S, Karapanos A, Coleman CI, Kluger J, White CM. Economic analysis of intravenous plus oral amiodarone, atrial septal pacing, and both strategies to prevent atrial fibrillation after open heart surgery. Pharmacotherapy. 2004;24(8):1013–9.CrossRef Reddy P, Kalus JS, Caron MF, Horowitz S, Karapanos A, Coleman CI, Kluger J, White CM. Economic analysis of intravenous plus oral amiodarone, atrial septal pacing, and both strategies to prevent atrial fibrillation after open heart surgery. Pharmacotherapy. 2004;24(8):1013–9.CrossRef
19.
go back to reference Foster EM, Jensen PS, Schlander M, Pelham WE Jr, Hechtman L, Arnold LE, Swanson JM, Wigal T. Treatment for ADHD: is more complex treatment cost-effective for more complex cases? Health Serv Res. 2007;42(1 Pt 1):165–82.CrossRef Foster EM, Jensen PS, Schlander M, Pelham WE Jr, Hechtman L, Arnold LE, Swanson JM, Wigal T. Treatment for ADHD: is more complex treatment cost-effective for more complex cases? Health Serv Res. 2007;42(1 Pt 1):165–82.CrossRef
20.
go back to reference Hollis JF, McAfee TA, Fellows JL, Zbikowski SM, Stark M, Riedlinger K. The effectiveness and cost effectiveness of telephone counselling and the nicotine patch in a state tobacco quitline. Tobacco Control. 2007;16(Supplement(1)):i53–9.CrossRef Hollis JF, McAfee TA, Fellows JL, Zbikowski SM, Stark M, Riedlinger K. The effectiveness and cost effectiveness of telephone counselling and the nicotine patch in a state tobacco quitline. Tobacco Control. 2007;16(Supplement(1)):i53–9.CrossRef
21.
go back to reference StataCorp: sureg — Zellner’s seemingly unrelated regression. In: Stata Base Reference Manual: Release 11. Volume Volume 3, Q-Z, edn. College Station: StataCorp LP; 2009: 2100–2107. StataCorp: sureg — Zellner’s seemingly unrelated regression. In: Stata Base Reference Manual: Release 11. Volume Volume 3, Q-Z, edn. College Station: StataCorp LP; 2009: 2100–2107.
22.
go back to reference Ng T. The impact of a preliminary test for interaction in a 2 x 2 factorial trial. Proc Biopharmaceutical Sec Am Stat Assoc Alexandria. 1991:220–7. Ng T. The impact of a preliminary test for interaction in a 2 x 2 factorial trial. Proc Biopharmaceutical Sec Am Stat Assoc Alexandria. 1991:220–7.
23.
go back to reference Simon R, Freedman LS. Bayesian design and analysis of two x two factorial clinical trials. Biometrics. 1997;53(2):456–64.CrossRef Simon R, Freedman LS. Bayesian design and analysis of two x two factorial clinical trials. Biometrics. 1997;53(2):456–64.CrossRef
24.
go back to reference Briggs A. Economic evaluation and clinical trials: size matters. Bmj. 2000;321(7273):1362–3.CrossRef Briggs A. Economic evaluation and clinical trials: size matters. Bmj. 2000;321(7273):1362–3.CrossRef
25.
go back to reference Backhouse ME. Use of randomised controlled trials for producing cost-effectiveness evidence: potential impact of design choices on sample size and study duration. Pharmacoeconomics. 2002;20(15):1061–77.CrossRef Backhouse ME. Use of randomised controlled trials for producing cost-effectiveness evidence: potential impact of design choices on sample size and study duration. Pharmacoeconomics. 2002;20(15):1061–77.CrossRef
26.
go back to reference Ramsey S, Willke R, Briggs A, Brown R, Buxton M, Chawla A, Cook J, Glick H, Liljas B, Petitti D, et al. Good research practices for cost-effectiveness analysis alongside clinical trials: the ISPOR RCT-CEA task force report. Value Health. 2005;8(5):521–33.CrossRef Ramsey S, Willke R, Briggs A, Brown R, Buxton M, Chawla A, Cook J, Glick H, Liljas B, Petitti D, et al. Good research practices for cost-effectiveness analysis alongside clinical trials: the ISPOR RCT-CEA task force report. Value Health. 2005;8(5):521–33.CrossRef
27.
go back to reference Hollingworth W, McKell-Redwood D, Hampson L, Metcalfe C: Cost utility analysis conducted alongside RCTs: is inefficiency considered in sample size calculations? Poster presented at the MRC Clinical Trials Methodology Conference, Bristol, UK, 4th–5th October 2011 2011. Hollingworth W, McKell-Redwood D, Hampson L, Metcalfe C: Cost utility analysis conducted alongside RCTs: is inefficiency considered in sample size calculations? Poster presented at the MRC Clinical Trials Methodology Conference, Bristol, UK, 4th–5th October 2011 2011.
28.
go back to reference Anderson P, Lloyd A. Outcomes research and health economics. In: Di Giovanna I, Hayes G, editors. Principles of Clinical Research. Stroud: Wrightson Biomedical Publishing Ltd; 2001. p. 291–316. Anderson P, Lloyd A. Outcomes research and health economics. In: Di Giovanna I, Hayes G, editors. Principles of Clinical Research. Stroud: Wrightson Biomedical Publishing Ltd; 2001. p. 291–316.
29.
go back to reference Marshall DA, Hux M. Design and analysis issues for economic analysis alongside clinical trials. Med Care. 2009;47(7 Suppl 1):S14–20.CrossRef Marshall DA, Hux M. Design and analysis issues for economic analysis alongside clinical trials. Med Care. 2009;47(7 Suppl 1):S14–20.CrossRef
30.
go back to reference O'Brien B. Economic evaluation of pharmaceuticals. Frankenstein's monster or vampire of trials? Med Care. 1996;34(12 Suppl):DS99–108.PubMed O'Brien B. Economic evaluation of pharmaceuticals. Frankenstein's monster or vampire of trials? Med Care. 1996;34(12 Suppl):DS99–108.PubMed
31.
go back to reference Byar DP. Factorial and reciprocal control designs. Stat Med. 1990;9(1–2):55–63 discussion 63-54.CrossRef Byar DP. Factorial and reciprocal control designs. Stat Med. 1990;9(1–2):55–63 discussion 63-54.CrossRef
32.
go back to reference Byar DP, Piantadosi S. Factorial designs for randomized clinical trials. Cancer Treat Rep. 1985;69(10):1055–63.PubMed Byar DP, Piantadosi S. Factorial designs for randomized clinical trials. Cancer Treat Rep. 1985;69(10):1055–63.PubMed
33.
go back to reference Yates F. Complex experiments (with discussion). J Roy Statist Soc. 1935;B2:181–247. Yates F. Complex experiments (with discussion). J Roy Statist Soc. 1935;B2:181–247.
34.
go back to reference Burnham KP, Anderson DR. Multimodel inference: understanding AIC and BIC in model selection. Sociol Methods Res. 2004;33(2):261–304.CrossRef Burnham KP, Anderson DR. Multimodel inference: understanding AIC and BIC in model selection. Sociol Methods Res. 2004;33(2):261–304.CrossRef
35.
go back to reference Lancelot R, Lesnoff M, McDermott JJ. Use of Akaike information criteria for model selection and inference: an application to assess prevention of gastrointestinal parasitism and respiratory mortality of Guinean goats in Kolda, Senegal. Preventive Veterinary Med. 2002;55(4):217–40.CrossRef Lancelot R, Lesnoff M, McDermott JJ. Use of Akaike information criteria for model selection and inference: an application to assess prevention of gastrointestinal parasitism and respiratory mortality of Guinean goats in Kolda, Senegal. Preventive Veterinary Med. 2002;55(4):217–40.CrossRef
36.
go back to reference Piantadosi S. Chapter 19: factorial trials. In: Clinical trials: a methodological perspective. 2nd ed. Hoboken: Wiley; 2005. p. 501–13.CrossRef Piantadosi S. Chapter 19: factorial trials. In: Clinical trials: a methodological perspective. 2nd ed. Hoboken: Wiley; 2005. p. 501–13.CrossRef
37.
go back to reference Frempong SN, Goranitis I, Oppong R. Economic evaluation alongside factorial trials: a systematic review of empirical studies. Expert Rev Pharmacoecon Outcomes Res. 2015;15(5):801–11.CrossRef Frempong SN, Goranitis I, Oppong R. Economic evaluation alongside factorial trials: a systematic review of empirical studies. Expert Rev Pharmacoecon Outcomes Res. 2015;15(5):801–11.CrossRef
38.
go back to reference Campbell AJ, Robertson MC, La Grow SJ, Kerse NM, Sanderson GF, Jacobs RJ, Sharp DM, Hale LA. Randomised controlled trial of prevention of falls in people aged >=75 with severe visual impairment: the VIP trial. BMJ. 2005;331(7520):817.CrossRef Campbell AJ, Robertson MC, La Grow SJ, Kerse NM, Sanderson GF, Jacobs RJ, Sharp DM, Hale LA. Randomised controlled trial of prevention of falls in people aged >=75 with severe visual impairment: the VIP trial. BMJ. 2005;331(7520):817.CrossRef
39.
go back to reference Sevick MA, Miller GD, Loeser RF, Williamson JD, Messier SP. Cost-effectiveness of exercise and diet in overweight and obese adults with knee osteoarthritis. Med Sci Sports Exerc. 2009;41(6):1167–74.CrossRef Sevick MA, Miller GD, Loeser RF, Williamson JD, Messier SP. Cost-effectiveness of exercise and diet in overweight and obese adults with knee osteoarthritis. Med Sci Sports Exerc. 2009;41(6):1167–74.CrossRef
40.
go back to reference Salize HJP, Merkel SD-R, Reinhard ID-M, Twardella DP, Mann KMD, Brenner HMDMPH. Cost-effective primary care-based strategies to improve smoking cessation: more value for money. Arch Intern Med. 2009;169(3):230–5.CrossRef Salize HJP, Merkel SD-R, Reinhard ID-M, Twardella DP, Mann KMD, Brenner HMDMPH. Cost-effective primary care-based strategies to improve smoking cessation: more value for money. Arch Intern Med. 2009;169(3):230–5.CrossRef
41.
go back to reference Lamb SE, Gates S, Underwood MR, Cooke MW, Ashby D, Szczepura A, Williams MA, Williamson EM, Withers EJ, Mt Isa S, et al. Managing injuries of the neck trial (MINT): design of a randomised controlled trial of treatments for whiplash associated disorders. BMC Musculoskelet Disord. 2007;8:7.CrossRef Lamb SE, Gates S, Underwood MR, Cooke MW, Ashby D, Szczepura A, Williams MA, Williamson EM, Withers EJ, Mt Isa S, et al. Managing injuries of the neck trial (MINT): design of a randomised controlled trial of treatments for whiplash associated disorders. BMC Musculoskelet Disord. 2007;8:7.CrossRef
42.
go back to reference Thorn JC, Ridyard CH, Hughes D, Wordsworth S, Mihaylova B, Noble S, Hollingworth W. Health economics analysis plans: where are we now? Value Health. 2016;19(7):A397.CrossRef Thorn JC, Ridyard CH, Hughes D, Wordsworth S, Mihaylova B, Noble S, Hollingworth W. Health economics analysis plans: where are we now? Value Health. 2016;19(7):A397.CrossRef
43.
go back to reference Dakin HA, Wordsworth S, Rogers CA, Abangma G, Raftery J, Harding SP, Lotery AJ, Downes SM, Chakravarthy U, Reeves BC, et al. Cost-effectiveness of ranibizumab and bevacizumab for age-related macular degeneration: 2-year findings from the IVAN randomised trial. BMJ Open. 2014;4(7):e005094.CrossRef Dakin HA, Wordsworth S, Rogers CA, Abangma G, Raftery J, Harding SP, Lotery AJ, Downes SM, Chakravarthy U, Reeves BC, et al. Cost-effectiveness of ranibizumab and bevacizumab for age-related macular degeneration: 2-year findings from the IVAN randomised trial. BMJ Open. 2014;4(7):e005094.CrossRef
Metadata
Title
Which interactions matter in economic evaluations? A systematic review and simulation study
Authors
Helen Dakin
Alastair Gray
Publication date
01-12-2020
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2020
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-020-00978-0

Other articles of this Issue 1/2020

BMC Medical Research Methodology 1/2020 Go to the issue