Skip to main content
Top
Published in: BMC Infectious Diseases 1/2022

Open Access 01-12-2022 | COVID-19 | Research

Chimeric forecasting: combining probabilistic predictions from computational models and human judgment

Authors: Thomas McAndrew, Allison Codi, Juan Cambeiro, Tamay Besiroglu, David Braun, Eva Chen, Luis Enrique Urtubey De Cèsaris, Damon Luk

Published in: BMC Infectious Diseases | Issue 1/2022

Login to get access

Abstract

Forecasts of the trajectory of an infectious agent can help guide public health decision making. A traditional approach to forecasting fits a computational model to structured data and generates a predictive distribution. However, human judgment has access to the same data as computational models plus experience, intuition, and subjective data. We propose a chimeric ensemble—a combination of computational and human judgment forecasts—as a novel approach to predicting the trajectory of an infectious agent. Each month from January, 2021 to June, 2021 we asked two generalist crowds, using the same criteria as the COVID-19 Forecast Hub, to submit a predictive distribution over incident cases and deaths at the US national level either two or three weeks into the future and combined these human judgment forecasts with forecasts from computational models submitted to the COVID-19 Forecasthub into a chimeric ensemble. We find a chimeric ensemble compared to an ensemble including only computational models improves predictions of incident cases and shows similar performance for predictions of incident deaths. A chimeric ensemble is a flexible, supportive public health tool and shows promising results for predictions of the spread of an infectious agent.
Appendix
Available only for authorised users
Literature
1.
go back to reference Lutz CS, Huynh MP, Schroeder M, Anyatonwu S, Dahlgren FS, Danyluk G, Fernandez D, Greene SK, Kipshidze N, Liu L, et al. Applying infectious disease forecasting to public health: a path forward using influenza forecasting examples. BMC Public Health. 2019;19(1):1–12.CrossRef Lutz CS, Huynh MP, Schroeder M, Anyatonwu S, Dahlgren FS, Danyluk G, Fernandez D, Greene SK, Kipshidze N, Liu L, et al. Applying infectious disease forecasting to public health: a path forward using influenza forecasting examples. BMC Public Health. 2019;19(1):1–12.CrossRef
2.
go back to reference Matthew B, Slayton RB, Johansson MA, Butler JC . Improving pandemic response: employing mathematical modeling to confront coronavirus disease 2019. Clin Infect Dis. 2021. Matthew B, Slayton RB, Johansson MA, Butler JC . Improving pandemic response: employing mathematical modeling to confront coronavirus disease 2019. Clin Infect Dis. 2021.
3.
go back to reference Matthew B, Cowling BJ, Cucunubá ZM, Dinh L, Ferguson NM, Gao H, Hill V, Imai N, Johansson MA, Kada S, et al. Early insights from statistical and mathematical modeling of key epidemiologic parameters of COVID-19. Emerg Infect Dis. 2020;26(11). Matthew B, Cowling BJ, Cucunubá ZM, Dinh L, Ferguson NM, Gao H, Hill V, Imai N, Johansson MA, Kada S, et al. Early insights from statistical and mathematical modeling of key epidemiologic parameters of COVID-19. Emerg Infect Dis. 2020;26(11).
6.
go back to reference Nikolopoulos K, Punia S, Schäfers A, Tsinopoulos C, Vasilakis C. Forecasting and planning during a pandemic: COVID-19 growth rates, supply chain disruptions, and governmental decisions. Eur J Oper Res. 2021;290(1):99–115.PubMedCrossRef Nikolopoulos K, Punia S, Schäfers A, Tsinopoulos C, Vasilakis C. Forecasting and planning during a pandemic: COVID-19 growth rates, supply chain disruptions, and governmental decisions. Eur J Oper Res. 2021;290(1):99–115.PubMedCrossRef
7.
go back to reference Borchering RK, Viboud C, Howerton E, Smith CP, Truelove S, Runge MC, Reich NG, Contamin L, Levander J, Salerno J, et al. Modeling of future COVID-19 cases, hospitalizations, and deaths, by vaccination rates and nonpharmaceutical intervention scenarios-United States, April–September 2021. Morb Mortal Wkly Rep. 2021;70(19):719.CrossRef Borchering RK, Viboud C, Howerton E, Smith CP, Truelove S, Runge MC, Reich NG, Contamin L, Levander J, Salerno J, et al. Modeling of future COVID-19 cases, hospitalizations, and deaths, by vaccination rates and nonpharmaceutical intervention scenarios-United States, April–September 2021. Morb Mortal Wkly Rep. 2021;70(19):719.CrossRef
8.
go back to reference Lu FS, Nguyen AT, Link NB, Davis JT, Chinazzi M, Xiong X, Vespignani A, Lipsitch M, Santillana M. Estimating the cumulative incidence of COVID-19 in the United States using four complementary approaches. medRxiv. 2020. Lu FS, Nguyen AT, Link NB, Davis JT, Chinazzi M, Xiong X, Vespignani A, Lipsitch M, Santillana M. Estimating the cumulative incidence of COVID-19 in the United States using four complementary approaches. medRxiv. 2020.
9.
go back to reference Doornik JA, Castle JL, Hendry DF. Short-term forecasting of the coronavirus pandemic. Int J Forecast. 2020. Doornik JA, Castle JL, Hendry DF. Short-term forecasting of the coronavirus pandemic. Int J Forecast. 2020.
10.
go back to reference Di Fusco M, Shea KM, Lin J, Nguyen JL, Angulo FJ, Benigno M, Malhotra D, Emir B, Sung AH, Hammond JL, et al. Health outcomes and economic burden of hospitalized COVID-19 patients in the United States. J Med Econ. 2021;24(1):308–17.PubMedCrossRef Di Fusco M, Shea KM, Lin J, Nguyen JL, Angulo FJ, Benigno M, Malhotra D, Emir B, Sung AH, Hammond JL, et al. Health outcomes and economic burden of hospitalized COVID-19 patients in the United States. J Med Econ. 2021;24(1):308–17.PubMedCrossRef
11.
go back to reference Simiao C, Klaus P, Michael K, Bloom DE. The economic burden of COVID-19 in the United States: estimates and projections under an infection-based herd immunity approach. J Econ Age. 2021;100328. Simiao C, Klaus P, Michael K, Bloom DE. The economic burden of COVID-19 in the United States: estimates and projections under an infection-based herd immunity approach. J Econ Age. 2021;100328.
12.
go back to reference Khullar D, Bond AM, Schpero WL. COVID-19 and the financial health of us hospitals. JAMA. 2020;323(21):2127–8.PubMedCrossRef Khullar D, Bond AM, Schpero WL. COVID-19 and the financial health of us hospitals. JAMA. 2020;323(21):2127–8.PubMedCrossRef
13.
go back to reference Kaufman BG, Whitaker R, Pink G, Holmes GM. Half of rural residents at high risk of serious illness due to COVID-19, creating stress on rural hospitals. J Rural Health. 2020;36(4):584–90.PubMedPubMedCentralCrossRef Kaufman BG, Whitaker R, Pink G, Holmes GM. Half of rural residents at high risk of serious illness due to COVID-19, creating stress on rural hospitals. J Rural Health. 2020;36(4):584–90.PubMedPubMedCentralCrossRef
14.
go back to reference Mehrotra P, Malani P, Yadav P. Personal protective equipment shortages during COVID-19-supply chain—related causes and mitigation strategies. JAMA Health Forum. 2020;1: e200553.PubMedCrossRef Mehrotra P, Malani P, Yadav P. Personal protective equipment shortages during COVID-19-supply chain—related causes and mitigation strategies. JAMA Health Forum. 2020;1: e200553.PubMedCrossRef
15.
go back to reference Wu H, Soe MM, Konnor R, Dantes R, Haass K, Dudeck MA, Gross C, Leaptrot D, Sapiano MRP, Allen-Bridson K, et al. Hospital capacities and shortages of healthcare resources among us hospitals during the coronavirus disease 2019 (COVID-19) pandemic, national healthcare safety network (NHSN), March 27–July 14, 2020. Infect Control Hosp Epidemiol. 2021; 1–4. Wu H, Soe MM, Konnor R, Dantes R, Haass K, Dudeck MA, Gross C, Leaptrot D, Sapiano MRP, Allen-Bridson K, et al. Hospital capacities and shortages of healthcare resources among us hospitals during the coronavirus disease 2019 (COVID-19) pandemic, national healthcare safety network (NHSN), March 27–July 14, 2020. Infect Control Hosp Epidemiol. 2021; 1–4.
16.
go back to reference Ray EL, Wattanachit N, Niemi J, Kanji AH, House K, Cramer EY, Bracher J, Zheng A, Yamana TK, Xiong X et al. Ensemble forecasts of coronavirus disease 2019 (COVID-19) in the us. MedRXiv. 2020. Ray EL, Wattanachit N, Niemi J, Kanji AH, House K, Cramer EY, Bracher J, Zheng A, Yamana TK, Xiong X et al. Ensemble forecasts of coronavirus disease 2019 (COVID-19) in the us. MedRXiv. 2020.
17.
go back to reference Bracher J, Wolffram D, Deuschel J, Görgen K, Ketterer JL, Ullrich A, Abbott S, Barbarossa MV, Bertsimas D, Bhatia S, et al. A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave. Nat Commun. 2021;12(1):1–16.CrossRef Bracher J, Wolffram D, Deuschel J, Görgen K, Ketterer JL, Ullrich A, Abbott S, Barbarossa MV, Bertsimas D, Bhatia S, et al. A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave. Nat Commun. 2021;12(1):1–16.CrossRef
18.
go back to reference European COVID-19 Forecast Hub. European COVID-19 Forecast Hub. European COVID-19 Forecast Hub. European COVID-19 Forecast Hub.
19.
go back to reference Cramer EY, Huang Y, Wang Y, Ray EL, Cornell M, Bracher J, Brennen A, Rivadeneira Alvaro JC, Gerding A, House K, et al. The United States COVID-19 forecast hub dataset. medRxiv. 2021. Cramer EY, Huang Y, Wang Y, Ray EL, Cornell M, Bracher J, Brennen A, Rivadeneira Alvaro JC, Gerding A, House K, et al. The United States COVID-19 forecast hub dataset. medRxiv. 2021.
20.
go back to reference Cramer EY, Lopez VK, Niemi J, George GE, Cegan JC , Dettwiller ID, England WP, Farthing MW, Hunter RH, Brandon L et al. Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US. medRxiv, 2021. Cramer EY, Lopez VK, Niemi J, George GE, Cegan JC , Dettwiller ID, England WP, Farthing MW, Hunter RH, Brandon L et al. Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US. medRxiv, 2021.
21.
go back to reference McAndrew T, Cambeiro J, Besiroglu T. Aggregating human judgment probabilistic predictions of the safety, efficacy, and timing of a COVID-19 vaccine. Vaccine. 2022;40(15):2331-41. McAndrew T, Cambeiro J, Besiroglu T. Aggregating human judgment probabilistic predictions of the safety, efficacy, and timing of a COVID-19 vaccine. Vaccine. 2022;40(15):2331-41.
22.
go back to reference Abbott S, Hellewell J, Thompson RN, Sherratt K, Gibbs HP, Bosse NI, Munday JD, Meakin S, Doughty EL, Chun JY, et al. Estimating the time-varying reproduction number of sars-cov-2 using national and subnational case counts. Wellcome Open Res. 2020;5(112):112.CrossRef Abbott S, Hellewell J, Thompson RN, Sherratt K, Gibbs HP, Bosse NI, Munday JD, Meakin S, Doughty EL, Chun JY, et al. Estimating the time-varying reproduction number of sars-cov-2 using national and subnational case counts. Wellcome Open Res. 2020;5(112):112.CrossRef
24.
go back to reference Ferstad JO, Gu AJ, Lee RY, Thapa I, Shin AY, Salomon JA, Glynn P, Shah NH, Milstein A, Schulman K, et al. A model to forecast regional demand for COVID-19 related hospital beds. MedRxiv, 2020. Ferstad JO, Gu AJ, Lee RY, Thapa I, Shin AY, Salomon JA, Glynn P, Shah NH, Milstein A, Schulman K, et al. A model to forecast regional demand for COVID-19 related hospital beds. MedRxiv, 2020.
25.
go back to reference Papastefanopoulos V, Linardatos P, Kotsiantis S. COVID-19: a comparison of time series methods to forecast percentage of active cases per population. Appl Sci. 2020;10(11):3880.CrossRef Papastefanopoulos V, Linardatos P, Kotsiantis S. COVID-19: a comparison of time series methods to forecast percentage of active cases per population. Appl Sci. 2020;10(11):3880.CrossRef
26.
go back to reference Maleki M, Mahmoudi MR, Wraith D, Pho K-H. Time series modelling to forecast the confirmed and recovered cases of COVID-19. Travel Med Infect Dis. 2020;37:101742.PubMedPubMedCentralCrossRef Maleki M, Mahmoudi MR, Wraith D, Pho K-H. Time series modelling to forecast the confirmed and recovered cases of COVID-19. Travel Med Infect Dis. 2020;37:101742.PubMedPubMedCentralCrossRef
27.
go back to reference Ingle TA, Morrison M, Wang X, Mercer T, Karman V, Fox S, Meyers LA. Projecting COVID-19 isolation bed requirements for people experiencing homelessness. PLoS ONE. 2021;16(5):e0251153.PubMedPubMedCentralCrossRef Ingle TA, Morrison M, Wang X, Mercer T, Karman V, Fox S, Meyers LA. Projecting COVID-19 isolation bed requirements for people experiencing homelessness. PLoS ONE. 2021;16(5):e0251153.PubMedPubMedCentralCrossRef
28.
go back to reference Janyce G, Kolawolé Valère S, Brezesky K, Romain Glèlè K. On the reliability of predictions on COVID-19 dynamics: a systematic and critical review of modelling techniques. Infect Dis Model. 2021. Janyce G, Kolawolé Valère S, Brezesky K, Romain Glèlè K. On the reliability of predictions on COVID-19 dynamics: a systematic and critical review of modelling techniques. Infect Dis Model. 2021.
29.
go back to reference Jewell NP, Lewnard JA, Jewell BL. Predictive mathematical models of underlying principles and value of the COVID-19 pandemic: projections. JAMA. 2020;323(19):1893–4.PubMedCrossRef Jewell NP, Lewnard JA, Jewell BL. Predictive mathematical models of underlying principles and value of the COVID-19 pandemic: projections. JAMA. 2020;323(19):1893–4.PubMedCrossRef
30.
go back to reference Meehan MT, Rojas DP, Adekunle AI, Adegboye OA, Caldwell JM, Turek E, Williams BM, Marais BJ, Trauer JM, McBryde ES. Modelling insights into the COVID-19 pandemic. Paediatr Respir Rev. 2020;35:64–9.PubMedPubMedCentral Meehan MT, Rojas DP, Adekunle AI, Adegboye OA, Caldwell JM, Turek E, Williams BM, Marais BJ, Trauer JM, McBryde ES. Modelling insights into the COVID-19 pandemic. Paediatr Respir Rev. 2020;35:64–9.PubMedPubMedCentral
31.
go back to reference Latif S, Usman M, Manzoor S, Iqbal W, Qadir J, Tyson G, Castro I, Razi A, Boulos MNK, Weller A, et al. Leveraging data science to combat COVID-19: a comprehensive review. IEEE Trans Artif Intell. 2020;1(1):85–103.CrossRef Latif S, Usman M, Manzoor S, Iqbal W, Qadir J, Tyson G, Castro I, Razi A, Boulos MNK, Weller A, et al. Leveraging data science to combat COVID-19: a comprehensive review. IEEE Trans Artif Intell. 2020;1(1):85–103.CrossRef
33.
go back to reference Arik SO, Li C-L, Yoon J, Sinha R, Epshteyn A, Le LT, Menon V, Singh S, Zhang L, Yoder N et al. Interpretable sequence learning for COVID-19 forecasting. arXiv preprint arXiv:2008.00646. 2020. Arik SO, Li C-L, Yoon J, Sinha R, Epshteyn A, Le LT, Menon V, Singh S, Zhang L, Yoder N et al. Interpretable sequence learning for COVID-19 forecasting. arXiv preprint arXiv:​2008.​00646. 2020.
34.
go back to reference Shinde GR, Kalamkar AB, Mahalle PN, Dey N, Chaki J, Hassanien AE. Forecasting models for coronavirus disease (COVID-19): a survey of the state-of-the-art. SN Comput Sci. 2020;1(4):1–15.CrossRef Shinde GR, Kalamkar AB, Mahalle PN, Dey N, Chaki J, Hassanien AE. Forecasting models for coronavirus disease (COVID-19): a survey of the state-of-the-art. SN Comput Sci. 2020;1(4):1–15.CrossRef
35.
go back to reference Mac S, Mishra S, Ximenes R, Barrett K, Khan YA, Naimark DMJ, Sander B. Modeling the coronavirus disease 2019 pandemic: a comprehensive guide of infectious disease and decision-analytic models. J Clin Epidemiol. 2021;132:133–41.PubMedCrossRef Mac S, Mishra S, Ximenes R, Barrett K, Khan YA, Naimark DMJ, Sander B. Modeling the coronavirus disease 2019 pandemic: a comprehensive guide of infectious disease and decision-analytic models. J Clin Epidemiol. 2021;132:133–41.PubMedCrossRef
36.
go back to reference Dimitrov NB, Meyers LA. Mathematical approaches to infectious disease prediction and control. In Risk and optimization in an uncertain world, pp 1–25. INFORMS, 2010. Dimitrov NB, Meyers LA. Mathematical approaches to infectious disease prediction and control. In Risk and optimization in an uncertain world, pp 1–25. INFORMS, 2010.
38.
go back to reference Wilson DJ. Weather, mobility, and COVID-19: a panel local projections estimator for understanding and forecasting infectious disease spread. Federal Reserve Bank of San Francisco, 2021. Wilson DJ. Weather, mobility, and COVID-19: a panel local projections estimator for understanding and forecasting infectious disease spread. Federal Reserve Bank of San Francisco, 2021.
39.
go back to reference Reinhart A, Brooks L, Jahja M, Rumack A, Tang J, Agrawal S, Al Saeed W, Arnold T, Basu A, Bien J, et al. An open repository of real-time COVID-19 indicators. Proc Natl Acad Sci. 2021;118(51). Reinhart A, Brooks L, Jahja M, Rumack A, Tang J, Agrawal S, Al Saeed W, Arnold T, Basu A, Bien J, et al. An open repository of real-time COVID-19 indicators. Proc Natl Acad Sci. 2021;118(51).
40.
go back to reference Lee M, Zhao J, Sun Q, Pan Y, Zhou W, Xiong C, Zhang L. Human mobility trends during the early stage of the COVID-19 pandemic in the United States. PLoS ONE. 2020;15(11): e0241468.PubMedPubMedCentralCrossRef Lee M, Zhao J, Sun Q, Pan Y, Zhou W, Xiong C, Zhang L. Human mobility trends during the early stage of the COVID-19 pandemic in the United States. PLoS ONE. 2020;15(11): e0241468.PubMedPubMedCentralCrossRef
41.
go back to reference Shen C, Chen A, Luo C, Zhang J, Feng B, Liao W, et al. Using reports of symptoms and diagnoses on social media to predict COVID-19 case counts in mainland china: Observational infoveillance study. J Med Internet Res. 2020;22(5): e19421.PubMedPubMedCentralCrossRef Shen C, Chen A, Luo C, Zhang J, Feng B, Liao W, et al. Using reports of symptoms and diagnoses on social media to predict COVID-19 case counts in mainland china: Observational infoveillance study. J Med Internet Res. 2020;22(5): e19421.PubMedPubMedCentralCrossRef
42.
43.
go back to reference Weiss HH. The sir model and the foundations of public health. Mater Math. 2013:0001–17 . Weiss HH. The sir model and the foundations of public health. Mater Math. 2013:0001–17 .
44.
go back to reference Kermack WO, McKendrick AG. A contribution to the mathematical theory of epidemics. Proc R Soc Ser A (containing papers of a London mathematical and physical character). 1927;115(772):700–21. Kermack WO, McKendrick AG. A contribution to the mathematical theory of epidemics. Proc R Soc Ser A (containing papers of a London mathematical and physical character). 1927;115(772):700–21.
45.
go back to reference Yang H-C, Xue Y, Pan Y, Liu Q, Hu G. Time fused coefficient sir model with application to COVID-19 epidemic in the United States. J Appl Stat. 2021:1–15. Yang H-C, Xue Y, Pan Y, Liu Q, Hu G. Time fused coefficient sir model with application to COVID-19 epidemic in the United States. J Appl Stat. 2021:1–15.
46.
go back to reference Chen Y-C, Ping-En L, Chang C-S, Liu T-H. A time-dependent sir model for COVID-19 with undetectable infected persons. IEEE Trans Netw Sci Eng. 2020;7(4):3279–94.CrossRef Chen Y-C, Ping-En L, Chang C-S, Liu T-H. A time-dependent sir model for COVID-19 with undetectable infected persons. IEEE Trans Netw Sci Eng. 2020;7(4):3279–94.CrossRef
47.
go back to reference Aleta A, Martin-Corral D, y Piontti AP, Ajelli M, Litvinova M, Chinazzi M, Dean NE, Halloran ME, Longini IM Jr, Merler S, et al. Modelling the impact of testing, contact tracing and household quarantine on second waves of COVID-19. Nat Hum Behav. 2020;4(9):964–71.PubMedPubMedCentralCrossRef Aleta A, Martin-Corral D, y Piontti AP, Ajelli M, Litvinova M, Chinazzi M, Dean NE, Halloran ME, Longini IM Jr, Merler S, et al. Modelling the impact of testing, contact tracing and household quarantine on second waves of COVID-19. Nat Hum Behav. 2020;4(9):964–71.PubMedPubMedCentralCrossRef
48.
go back to reference Gibson GC, Reich NG, Sheldon D. Real-time mechanistic Bayesian forecasts of COVID-19 mortality. medRxiv. 2020. Gibson GC, Reich NG, Sheldon D. Real-time mechanistic Bayesian forecasts of COVID-19 mortality. medRxiv. 2020.
49.
go back to reference Liu M, Thomadsen R, Yao S. Forecasting the spread of COVID-19 under different reopening strategies. Sci Rep. 2020;10(1):1–8. Liu M, Thomadsen R, Yao S. Forecasting the spread of COVID-19 under different reopening strategies. Sci Rep. 2020;10(1):1–8.
50.
go back to reference Fowler JH, Hill S, Levin R, Obradovich N. The effect of stay-at-home orders on COVID-19 infections in the United States. Available at SSRN 3576826. 2020. Fowler JH, Hill S, Levin R, Obradovich N. The effect of stay-at-home orders on COVID-19 infections in the United States. Available at SSRN 3576826. 2020.
51.
52.
go back to reference Kudryashov NA, Chmykhov MA, Vigdorowitsch M. Analytical features of the sir model and their applications to COVID-19. Appl Math Model. 2021;90:466–73.PubMedCrossRef Kudryashov NA, Chmykhov MA, Vigdorowitsch M. Analytical features of the sir model and their applications to COVID-19. Appl Math Model. 2021;90:466–73.PubMedCrossRef
53.
go back to reference Chinazzi M, Davis JT, Ajelli M, Gioannini C, Litvinova M, Merler S, yPiontti AP, Kunpeng M, Rossi L, Sun K, et al. The effect of travel restrictions on the spread of the 2019 novel coronavirus (COVID-19) outbreak. Science. 2020;368(6489):395–400.PubMedPubMedCentralCrossRef Chinazzi M, Davis JT, Ajelli M, Gioannini C, Litvinova M, Merler S, yPiontti AP, Kunpeng M, Rossi L, Sun K, et al. The effect of travel restrictions on the spread of the 2019 novel coronavirus (COVID-19) outbreak. Science. 2020;368(6489):395–400.PubMedPubMedCentralCrossRef
54.
go back to reference Ritterman J, Osborne M, Klein E. Using prediction markets and twitter to predict a swine flu pandemic. In 1st international workshop on mining social media. 2009;9:9–17. Ritterman J, Osborne M, Klein E. Using prediction markets and twitter to predict a swine flu pandemic. In 1st international workshop on mining social media. 2009;9:9–17.
55.
go back to reference Li EY, Tung C-Y, Chang S-H. The wisdom of crowds in action: forecasting epidemic diseases with a web-based prediction market system. Int J Med Inform. 2016;92:35–43.PubMedCrossRef Li EY, Tung C-Y, Chang S-H. The wisdom of crowds in action: forecasting epidemic diseases with a web-based prediction market system. Int J Med Inform. 2016;92:35–43.PubMedCrossRef
56.
go back to reference Tung C, Chou T-C, Lin J. Using prediction markets of market scoring rule to forecast infectious diseases: a case study in Taiwan. BMC Public Health. 2015;15(1):1–12.CrossRef Tung C, Chou T-C, Lin J. Using prediction markets of market scoring rule to forecast infectious diseases: a case study in Taiwan. BMC Public Health. 2015;15(1):1–12.CrossRef
57.
go back to reference Wolfers J, Zitzewitz E. Prediction markets. J Econ Perspect. 2004;18(2):107–26.CrossRef Wolfers J, Zitzewitz E. Prediction markets. J Econ Perspect. 2004;18(2):107–26.CrossRef
58.
go back to reference Kambil A, Van Heck E. Making markets: how firms can design and profit from online auctions and exchanges. Harvard: Harvard Business School Press; 2002. Kambil A, Van Heck E. Making markets: how firms can design and profit from online auctions and exchanges. Harvard: Harvard Business School Press; 2002.
59.
go back to reference McHugh P, Jackson AL. Prediction market accuracy: the impact of size, incentives, context and interpretation. J Predict Markets. 2012;6(2). McHugh P, Jackson AL. Prediction market accuracy: the impact of size, incentives, context and interpretation. J Predict Markets. 2012;6(2).
60.
go back to reference Samaras L, García-Barriocanal E, Sicilia M-A. Comparing social media and google to detect and predict severe epidemics. Sci Rep. 2020;10(1):1–11.CrossRef Samaras L, García-Barriocanal E, Sicilia M-A. Comparing social media and google to detect and predict severe epidemics. Sci Rep. 2020;10(1):1–11.CrossRef
61.
go back to reference Al-Garadi MA, Khan MS, Varathan KD, Mujtaba G, Al-Kabsi AM. Using online social networks to track a pandemic: a systematic review. J Biomed Inform. 2016;62:1–11.PubMedCrossRef Al-Garadi MA, Khan MS, Varathan KD, Mujtaba G, Al-Kabsi AM. Using online social networks to track a pandemic: a systematic review. J Biomed Inform. 2016;62:1–11.PubMedCrossRef
62.
go back to reference Alessa A, Faezipour M. A review of influenza detection and prediction through social networking sites. Theor Biol Med Model. 2018;15(1):1–27.CrossRef Alessa A, Faezipour M. A review of influenza detection and prediction through social networking sites. Theor Biol Med Model. 2018;15(1):1–27.CrossRef
63.
go back to reference Masri S, Jia J, Li C, Zhou G, Lee M-C, Yan G, Jun W. Use of twitter data to improve zika virus surveillance in the United States during the 2016 epidemic. BMC Public Health. 2019;19(1):1–14.CrossRef Masri S, Jia J, Li C, Zhou G, Lee M-C, Yan G, Jun W. Use of twitter data to improve zika virus surveillance in the United States during the 2016 epidemic. BMC Public Health. 2019;19(1):1–14.CrossRef
64.
go back to reference Marques-Toledo CA, Degener CM, Vinhal L, Coelho G, Meira W, Codeço CT, Teixeira MM. Dengue prediction by the web: Tweets are a useful tool for estimating and forecasting dengue at country and city level. PLoS Negl Trop Dis. 2017;11(7):e0005729.PubMedPubMedCentralCrossRef Marques-Toledo CA, Degener CM, Vinhal L, Coelho G, Meira W, Codeço CT, Teixeira MM. Dengue prediction by the web: Tweets are a useful tool for estimating and forecasting dengue at country and city level. PLoS Negl Trop Dis. 2017;11(7):e0005729.PubMedPubMedCentralCrossRef
65.
go back to reference Ning S, Yang S, Kou SC. Accurate regional influenza epidemics tracking using internet search data. Sci Rep. 2019;9(1):1–8.CrossRef Ning S, Yang S, Kou SC. Accurate regional influenza epidemics tracking using internet search data. Sci Rep. 2019;9(1):1–8.CrossRef
66.
go back to reference McAndrew T, Reich NG. An expert judgment model to predict early stages of the COVID-19 pandemic in the United States. PLoS Comput Biol. 2022;18(9);e1010485. McAndrew T, Reich NG. An expert judgment model to predict early stages of the COVID-19 pandemic in the United States. PLoS Comput Biol. 2022;18(9);e1010485.
67.
go back to reference Farrow DC, Brooks LC, Hyun S, Tibshirani RJ, Burke DS, Rosenfeld R. A human judgment approach to epidemiological forecasting. PLoS Comput Biol. 2017;13(3):e1005248.PubMedPubMedCentralCrossRef Farrow DC, Brooks LC, Hyun S, Tibshirani RJ, Burke DS, Rosenfeld R. A human judgment approach to epidemiological forecasting. PLoS Comput Biol. 2017;13(3):e1005248.PubMedPubMedCentralCrossRef
68.
go back to reference Bosse NI, Abbott S, Bracher J, Hain H, Quilty BJ, Jit M, van Leeuwen E, Cori A, Funk S, et al. Comparing human and model-based forecasts of COVID-19 in Germany and Poland. medRxiv, 2021. Bosse NI, Abbott S, Bracher J, Hain H, Quilty BJ, Jit M, van Leeuwen E, Cori A, Funk S, et al. Comparing human and model-based forecasts of COVID-19 in Germany and Poland. medRxiv, 2021.
69.
70.
go back to reference Hanea AM, Hemming V, Nane GF. Uncertainty quantification with experts: present status and research needs. Risk Anal. 2021. Hanea AM, Hemming V, Nane GF. Uncertainty quantification with experts: present status and research needs. Risk Anal. 2021.
71.
go back to reference Clemen RT. Combining forecasts: a review and annotated bibliography. Int J Forecast. 1989;5(4):559–83.CrossRef Clemen RT. Combining forecasts: a review and annotated bibliography. Int J Forecast. 1989;5(4):559–83.CrossRef
72.
go back to reference Bates JM, Granger CWJ. The combination of forecasts. J Oper Res Soc. 1969;20(4):451–68.CrossRef Bates JM, Granger CWJ. The combination of forecasts. J Oper Res Soc. 1969;20(4):451–68.CrossRef
73.
go back to reference Clemen RT, Winkler RL. Combining probability distributions from experts in risk analysis. Risk Anal. 1999;19(2):187–203.CrossRef Clemen RT, Winkler RL. Combining probability distributions from experts in risk analysis. Risk Anal. 1999;19(2):187–203.CrossRef
74.
go back to reference Winkler RL. Combining forecasts: a philosophical basis and some current issues. Int J Forecast. 1989;5(4):605–9.CrossRef Winkler RL. Combining forecasts: a philosophical basis and some current issues. Int J Forecast. 1989;5(4):605–9.CrossRef
75.
go back to reference Genest C, McConway KJ. Allocating the weights in the linear opinion pool. J Forecast. 1990;9(1):53–73.CrossRef Genest C, McConway KJ. Allocating the weights in the linear opinion pool. J Forecast. 1990;9(1):53–73.CrossRef
76.
go back to reference Blattberg RC, Hoch SJ. Database models and managerial intuition: 50% model+ 50% manager. In Perspectives on promotion and database marketing: the collected works of Robert C Blattberg. pp 215–227. World Scientific, 2010. Blattberg RC, Hoch SJ. Database models and managerial intuition: 50% model+ 50% manager. In Perspectives on promotion and database marketing: the collected works of Robert C Blattberg. pp 215–227. World Scientific, 2010.
77.
go back to reference Dane E, Pratt MG. Exploring intuition and its role in managerial decision making. Acad Manag Rev. 2007;32(1):33–54.CrossRef Dane E, Pratt MG. Exploring intuition and its role in managerial decision making. Acad Manag Rev. 2007;32(1):33–54.CrossRef
78.
go back to reference Armstrong JS. Combining forecasts. In Principles of forecasting. Springer; 2001, . pp 417–439. Armstrong JS. Combining forecasts. In Principles of forecasting. Springer; 2001, . pp 417–439.
79.
go back to reference Lawrence M, Goodwin P, O’Connor M, Önkal D. Judgmental forecasting: a review of progress over the last 25 years. Int J Forecast. 2006;22(3):493–518.CrossRef Lawrence M, Goodwin P, O’Connor M, Önkal D. Judgmental forecasting: a review of progress over the last 25 years. Int J Forecast. 2006;22(3):493–518.CrossRef
80.
go back to reference Sanders NR, Ritzman LP. Bringing judgment into combination forecasts. J Oper Manag. 1995;13(4):311–21.CrossRef Sanders NR, Ritzman LP. Bringing judgment into combination forecasts. J Oper Manag. 1995;13(4):311–21.CrossRef
81.
go back to reference Whitecotton SM, Sanders DE, Norris KB. Improving predictive accuracy with a combination of human intuition and mechanical decision aids. Organ Behav Hum Decis Process. 1998;76(3):325–48.PubMedCrossRef Whitecotton SM, Sanders DE, Norris KB. Improving predictive accuracy with a combination of human intuition and mechanical decision aids. Organ Behav Hum Decis Process. 1998;76(3):325–48.PubMedCrossRef
82.
go back to reference Goodwin P. Correct or combine? mechanically integrating judgmental forecasts with statistical methods. Int J Forecast. 2000;16(2):261–75.CrossRef Goodwin P. Correct or combine? mechanically integrating judgmental forecasts with statistical methods. Int J Forecast. 2000;16(2):261–75.CrossRef
83.
go back to reference Webby R, O’Connor M. Judgemental and statistical time series forecasting: a review of the literature. Int J Forecast. 1996;12(1):91–118.CrossRef Webby R, O’Connor M. Judgemental and statistical time series forecasting: a review of the literature. Int J Forecast. 1996;12(1):91–118.CrossRef
84.
go back to reference Petropoulos F, Kourentzes N, Nikolopoulos K, Siemsen E. Judgmental selection of forecasting models. J Oper Manag. 2018;60:34–46.CrossRef Petropoulos F, Kourentzes N, Nikolopoulos K, Siemsen E. Judgmental selection of forecasting models. J Oper Manag. 2018;60:34–46.CrossRef
85.
go back to reference Albuquerque Pedro HM, Peng Y, Fontoura da Silva JP. Making the whole greater than the sum of its parts: a literature review of ensemble methods for financial time series forecasting. J Forecast. 2022. Albuquerque Pedro HM, Peng Y, Fontoura da Silva JP. Making the whole greater than the sum of its parts: a literature review of ensemble methods for financial time series forecasting. J Forecast. 2022.
86.
go back to reference Cramer EY, Huang Y, Wang Y, Ray EL, Cornell M, Bracher J, Brennen A, Castro Rivadeneira AJ, Aaron G, Katie H, Dasuni J, Kanji AH, Ayush K, Khoa L, Jarad N, Ariane S, Apurv S, Nutcha W, Zorn MW, Reich NG, US COVID-19 Forecast Hub Consortium. The United States COVID-19 forecast hub dataset. medRxiv, 2021. Cramer EY, Huang Y, Wang Y, Ray EL, Cornell M, Bracher J, Brennen A, Castro Rivadeneira AJ, Aaron G, Katie H, Dasuni J, Kanji AH, Ayush K, Khoa L, Jarad N, Ariane S, Apurv S, Nutcha W, Zorn MW, Reich NG, US COVID-19 Forecast Hub Consortium. The United States COVID-19 forecast hub dataset. medRxiv, 2021.
88.
go back to reference Gneiting T, Raftery AE. Strictly proper scoring rules, prediction, and estimation. J Am Stat Assoc. 2007;102(477):359–78.CrossRef Gneiting T, Raftery AE. Strictly proper scoring rules, prediction, and estimation. J Am Stat Assoc. 2007;102(477):359–78.CrossRef
89.
go back to reference Gneiting T, Balabdaoui F, Raftery AE. Probabilistic forecasts, calibration and sharpness. J R Stat Soc Ser B (Stat Methodol). 2007;69(2):243–68.CrossRef Gneiting T, Balabdaoui F, Raftery AE. Probabilistic forecasts, calibration and sharpness. J R Stat Soc Ser B (Stat Methodol). 2007;69(2):243–68.CrossRef
90.
91.
go back to reference Storn R, Price K. Differential evolution—a simple and efficient heuristic for global optimization over continuous spaces. J Global Optim. 1997;11(4):341–59.CrossRef Storn R, Price K. Differential evolution—a simple and efficient heuristic for global optimization over continuous spaces. J Global Optim. 1997;11(4):341–59.CrossRef
92.
go back to reference McKerns MM, Strand L, Sullivan T, Fang A, Aivazis MAG. Building a framework for predictive science. arXiv preprint arXiv:1202.1056, 2012. McKerns MM, Strand L, Sullivan T, Fang A, Aivazis MAG. Building a framework for predictive science. arXiv preprint arXiv:​1202.​1056, 2012.
93.
go back to reference McKerns M, Hung P, Aivazis M. mystic: highly-constrained non-convex optimization and uq, 2009. McKerns M, Hung P, Aivazis M. mystic: highly-constrained non-convex optimization and uq, 2009.
94.
go back to reference Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, et al. Scikit-learn machine learning in python. J Mach Learn Res. 2011;12:2825–30. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, et al. Scikit-learn machine learning in python. J Mach Learn Res. 2011;12:2825–30.
95.
go back to reference Reich NG, Brooks LC, Fox SJ, Kandula S, McGowan CJ, Moore E, Osthus D, Ray EL, Tushar A, Yamana TK, et al. A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States. Proc Natl Acad Sci. 2019;116(8):3146–54.PubMedPubMedCentralCrossRef Reich NG, Brooks LC, Fox SJ, Kandula S, McGowan CJ, Moore E, Osthus D, Ray EL, Tushar A, Yamana TK, et al. A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States. Proc Natl Acad Sci. 2019;116(8):3146–54.PubMedPubMedCentralCrossRef
96.
go back to reference McGowan CJ, Biggerstaff M, Johansson M, Apfeldorf KM, Ben-Nun M, Brooks L, Convertino M, Erraguntla M, Farrow DC, Freeze J, et al. Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016. Sci Rep. 2019;9(1):1–13.CrossRef McGowan CJ, Biggerstaff M, Johansson M, Apfeldorf KM, Ben-Nun M, Brooks L, Convertino M, Erraguntla M, Farrow DC, Freeze J, et al. Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016. Sci Rep. 2019;9(1):1–13.CrossRef
97.
go back to reference McCandless TC, Haupt SE, Young GS. The effects of imputing missing data on ensemble temperature forecasts. J Comput. 2011;6(2):162–71.CrossRef McCandless TC, Haupt SE, Young GS. The effects of imputing missing data on ensemble temperature forecasts. J Comput. 2011;6(2):162–71.CrossRef
98.
go back to reference McAndrew T, Reich NG. Adaptively stacking ensembles for influenza forecasting. Stat Med. 2021. McAndrew T, Reich NG. Adaptively stacking ensembles for influenza forecasting. Stat Med. 2021.
99.
go back to reference Merkle EC, Saw G, Davis-Stober C. Beating the average forecast: regularization based on forecaster attributes. J Math Psychol. 2020;98: 102419.CrossRef Merkle EC, Saw G, Davis-Stober C. Beating the average forecast: regularization based on forecaster attributes. J Math Psychol. 2020;98: 102419.CrossRef
100.
go back to reference Lagnado DA, Newell BR, Kahan S, Shanks DR. Insight and strategy in multiple-cue learning. J Exp Psychol Gen. 2006;135(2):162.PubMedCrossRef Lagnado DA, Newell BR, Kahan S, Shanks DR. Insight and strategy in multiple-cue learning. J Exp Psychol Gen. 2006;135(2):162.PubMedCrossRef
101.
go back to reference Hammond KR, Summers DA. Cognitive dependence on linear and nonlinear cues. Psychol Rev. 1965;72(3):215.PubMedCrossRef Hammond KR, Summers DA. Cognitive dependence on linear and nonlinear cues. Psychol Rev. 1965;72(3):215.PubMedCrossRef
102.
go back to reference Spicer S, Mitchell C, Wills A, Blake K, Jones P. Theory protection: do humans protect existing associative links? 2021. Spicer S, Mitchell C, Wills A, Blake K, Jones P. Theory protection: do humans protect existing associative links? 2021.
Metadata
Title
Chimeric forecasting: combining probabilistic predictions from computational models and human judgment
Authors
Thomas McAndrew
Allison Codi
Juan Cambeiro
Tamay Besiroglu
David Braun
Eva Chen
Luis Enrique Urtubey De Cèsaris
Damon Luk
Publication date
01-12-2022
Publisher
BioMed Central
Keyword
COVID-19
Published in
BMC Infectious Diseases / Issue 1/2022
Electronic ISSN: 1471-2334
DOI
https://doi.org/10.1186/s12879-022-07794-5

Other articles of this Issue 1/2022

BMC Infectious Diseases 1/2022 Go to the issue
Obesity Clinical Trial Summary

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.

Developed by: Springer Medicine

Highlights from the ACC 2024 Congress

Year in Review: Pediatric cardiology

Watch Dr. Anne Marie Valente present the last year's highlights in pediatric and congenital heart disease in the official ACC.24 Year in Review session.

Year in Review: Pulmonary vascular disease

The last year's highlights in pulmonary vascular disease are presented by Dr. Jane Leopold in this official video from ACC.24.

Year in Review: Valvular heart disease

Watch Prof. William Zoghbi present the last year's highlights in valvular heart disease from the official ACC.24 Year in Review session.

Year in Review: Heart failure and cardiomyopathies

Watch this official video from ACC.24. Dr. Biykem Bozkurt discusses last year's major advances in heart failure and cardiomyopathies.