Skip to main content
Top
Published in: BMC Medical Research Methodology 1/2020

Open Access 01-12-2020 | SARS-CoV-2 | Research article

Predictive accuracy of a hierarchical logistic model of cumulative SARS-CoV-2 case growth until May 2020

Author: Levente Kriston

Published in: BMC Medical Research Methodology | Issue 1/2020

Login to get access

Abstract

Background

Infectious disease predictions models, including virtually all epidemiological models describing the spread of the SARS-CoV-2 pandemic, are rarely evaluated empirically. The aim of the present study was to investigate the predictive accuracy of a prognostic model for forecasting the development of the cumulative number of reported SARS-CoV-2 cases in countries and administrative regions worldwide until the end of May 2020.

Methods

The cumulative number of reported SARS-CoV-2 cases was forecasted in 251 regions with a horizon of two weeks, one month, and two months using a hierarchical logistic model at the end of March 2020. Forecasts were compared to actual observations by using a series of evaluation metrics.

Results

On average, predictive accuracy was very high in nearly all regions at the two weeks forecast, high in most regions at the one month forecast, and notable in the majority of the regions at the two months forecast. Higher accuracy was associated with the availability of more data for estimation and with a more pronounced cumulative case growth from the first case to the date of estimation. In some strongly affected regions, cumulative case counts were considerably underestimated.

Conclusions

With keeping its limitations in mind, the investigated model may be used for the preparation and distribution of resources during the initial phase of epidemics. Future research should primarily address the model’s assumptions and its scope of applicability. In addition, establishing a relationship with known mechanisms and traditional epidemiological models of disease transmission would be desirable.
Literature
1.
go back to reference Heesterbeek H, Anderson RM, Andreasen V, Bansal S, De Angelis D, Dye C, et al. Modeling infectious disease dynamics in the complex landscape of global health. Science. 2015;347:aaa4339.CrossRef Heesterbeek H, Anderson RM, Andreasen V, Bansal S, De Angelis D, Dye C, et al. Modeling infectious disease dynamics in the complex landscape of global health. Science. 2015;347:aaa4339.CrossRef
2.
go back to reference Holmdahl I, Buckee C. Wrong but useful - what Covid-19 epidemiologic models can and cannot tell us. N Engl J Med. 2020;383:303–5.CrossRef Holmdahl I, Buckee C. Wrong but useful - what Covid-19 epidemiologic models can and cannot tell us. N Engl J Med. 2020;383:303–5.CrossRef
3.
go back to reference Adam D. Special report: the simulations driving the world’s response to COVID-19. Nature. 2020;580:316–8.CrossRef Adam D. Special report: the simulations driving the world’s response to COVID-19. Nature. 2020;580:316–8.CrossRef
5.
go back to reference Li S-L, Bjørnstad ON, Ferrari MJ, Mummah R, Runge MC, Fonnesbeck CJ, et al. Essential information: uncertainty and optimal control of Ebola outbreaks. PNAS. 2017;114:5659–64.CrossRef Li S-L, Bjørnstad ON, Ferrari MJ, Mummah R, Runge MC, Fonnesbeck CJ, et al. Essential information: uncertainty and optimal control of Ebola outbreaks. PNAS. 2017;114:5659–64.CrossRef
6.
go back to reference Probert WJM, Jewell CP, Werkman M, Fonnesbeck CJ, Goto Y, Runge MC, et al. Real-time decision-making during emergency disease outbreaks. PLoS Comput Biol. 2018;14:e1006202.CrossRef Probert WJM, Jewell CP, Werkman M, Fonnesbeck CJ, Goto Y, Runge MC, et al. Real-time decision-making during emergency disease outbreaks. PLoS Comput Biol. 2018;14:e1006202.CrossRef
7.
go back to reference Funk S, Camacho A, Kucharski AJ, Lowe R, Eggo RM, Edmunds WJ. Assessing the performance of real-time epidemic forecasts: a case study of Ebola in the Western area region of Sierra Leone, 2014-15. PLoS Comput Biol. 2019;15:e1006785.CrossRef Funk S, Camacho A, Kucharski AJ, Lowe R, Eggo RM, Edmunds WJ. Assessing the performance of real-time epidemic forecasts: a case study of Ebola in the Western area region of Sierra Leone, 2014-15. PLoS Comput Biol. 2019;15:e1006785.CrossRef
8.
go back to reference Johansson MA, Reich NG, Hota A, Brownstein JS, Santillana M. Evaluating the performance of infectious disease forecasts: a comparison of climate-driven and seasonal dengue forecasts for Mexico. Sci Rep. 2016;6:33707. Johansson MA, Reich NG, Hota A, Brownstein JS, Santillana M. Evaluating the performance of infectious disease forecasts: a comparison of climate-driven and seasonal dengue forecasts for Mexico. Sci Rep. 2016;6:33707.
9.
go back to reference Hsieh Y-H, Cheng Y-S. Real-time forecast of multiphase outbreak. Emerg Infect Dis. 2006;12:122–7.CrossRef Hsieh Y-H, Cheng Y-S. Real-time forecast of multiphase outbreak. Emerg Infect Dis. 2006;12:122–7.CrossRef
11.
go back to reference Biggerstaff M, Alper D, Dredze M, Fox S, Fung IC-H, Hickmann KS, et al. Results from the centers for disease control and prevention’s predict the 2013–2014 influenza season challenge. BMC Infect Dis. 2016;16:357.CrossRef Biggerstaff M, Alper D, Dredze M, Fox S, Fung IC-H, Hickmann KS, et al. Results from the centers for disease control and prevention’s predict the 2013–2014 influenza season challenge. BMC Infect Dis. 2016;16:357.CrossRef
12.
go back to reference Hsieh Y-H, Fisman DN, Wu J. On epidemic modeling in real time: an application to the 2009 novel a (H1N1) influenza outbreak in Canada. BMC Res Notes. 2010;3:283.CrossRef Hsieh Y-H, Fisman DN, Wu J. On epidemic modeling in real time: an application to the 2009 novel a (H1N1) influenza outbreak in Canada. BMC Res Notes. 2010;3:283.CrossRef
13.
go back to reference Chowell G, Viboud C, Simonsen L, Merler S, Vespignani A. Perspectives on model forecasts of the 2014–2015 Ebola epidemic in West Africa: lessons and the way forward. BMC Med. 2017;15:42.CrossRef Chowell G, Viboud C, Simonsen L, Merler S, Vespignani A. Perspectives on model forecasts of the 2014–2015 Ebola epidemic in West Africa: lessons and the way forward. BMC Med. 2017;15:42.CrossRef
14.
go back to reference Pell B, Kuang Y, Viboud C, Chowell G. Using phenomenological models for forecasting the 2015 Ebola challenge. Epidemics. 2018;22:62–70.CrossRef Pell B, Kuang Y, Viboud C, Chowell G. Using phenomenological models for forecasting the 2015 Ebola challenge. Epidemics. 2018;22:62–70.CrossRef
15.
go back to reference Reich NG, Lauer SA, Sakrejda K, Iamsirithaworn S, Hinjoy S, Suangtho P, et al. Challenges in real-time prediction of infectious disease: a case study of dengue in Thailand. PLoS Negl Trop Dis. 2016;10:e0004761.CrossRef Reich NG, Lauer SA, Sakrejda K, Iamsirithaworn S, Hinjoy S, Suangtho P, et al. Challenges in real-time prediction of infectious disease: a case study of dengue in Thailand. PLoS Negl Trop Dis. 2016;10:e0004761.CrossRef
16.
go back to reference Liu F, Porco TC, Amza A, Kadri B, Nassirou B, West SK, et al. Short-term forecasting of the prevalence of trachoma: expert opinion, statistical regression, versus transmission models. PLoS Negl Trop Dis. 2015;9:e0004000.CrossRef Liu F, Porco TC, Amza A, Kadri B, Nassirou B, West SK, et al. Short-term forecasting of the prevalence of trachoma: expert opinion, statistical regression, versus transmission models. PLoS Negl Trop Dis. 2015;9:e0004000.CrossRef
18.
go back to reference Hellewell J, Abbott S, Gimma A, Bosse NI, Jarvis CI, Russell TW, et al. Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts. Lancet Glob Health. 2020;8:e488–96.CrossRef Hellewell J, Abbott S, Gimma A, Bosse NI, Jarvis CI, Russell TW, et al. Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts. Lancet Glob Health. 2020;8:e488–96.CrossRef
19.
go back to reference Kucharski AJ, Russell TW, Diamond C, Liu Y, Edmunds J, Funk S, et al. Early dynamics of transmission and control of COVID-19: a mathematical modelling study. Lancet Infect Dis. 2020;20:553–8.CrossRef Kucharski AJ, Russell TW, Diamond C, Liu Y, Edmunds J, Funk S, et al. Early dynamics of transmission and control of COVID-19: a mathematical modelling study. Lancet Infect Dis. 2020;20:553–8.CrossRef
20.
go back to reference Wu JT, Leung K, Leung GM. Nowcasting and forecasting the potential domestic and international spread of the 2019-nCoV outbreak originating in Wuhan, China: a modelling study. Lancet. 2020;395:689–97.CrossRef Wu JT, Leung K, Leung GM. Nowcasting and forecasting the potential domestic and international spread of the 2019-nCoV outbreak originating in Wuhan, China: a modelling study. Lancet. 2020;395:689–97.CrossRef
21.
go back to reference Koo JR, Cook AR, Park M, Sun Y, Sun H, Lim JT, et al. Interventions to mitigate early spread of SARS-CoV-2 in Singapore: a modelling study. Lancet Infect Dis. 2020;20:678–88.CrossRef Koo JR, Cook AR, Park M, Sun Y, Sun H, Lim JT, et al. Interventions to mitigate early spread of SARS-CoV-2 in Singapore: a modelling study. Lancet Infect Dis. 2020;20:678–88.CrossRef
22.
go back to reference Roosa K, Lee Y, Luo R, Kirpich A, Rothenberg R, Hyman JM, et al. Real-time forecasts of the COVID-19 epidemic in China from February 5th to February 24th, 2020. Infect Dis Model. 2020;5:256–63.PubMedPubMedCentral Roosa K, Lee Y, Luo R, Kirpich A, Rothenberg R, Hyman JM, et al. Real-time forecasts of the COVID-19 epidemic in China from February 5th to February 24th, 2020. Infect Dis Model. 2020;5:256–63.PubMedPubMedCentral
23.
go back to reference IHME COVID-19 health service utilization forecasting team, Murray CJ. Forecasting COVID-19 impact on hospital bed-days, ICU-days, ventilator-days and deaths by US state in the next 4 months. medRxiv. 2020; 2020.03.27.20043752. IHME COVID-19 health service utilization forecasting team, Murray CJ. Forecasting COVID-19 impact on hospital bed-days, ICU-days, ventilator-days and deaths by US state in the next 4 months. medRxiv. 2020; 2020.03.27.20043752.
25.
go back to reference Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis. 2020;20:533–4.CrossRef Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis. 2020;20:533–4.CrossRef
27.
go back to reference Kingsland S. The refractory model: the logistic curve and the history of population ecology. Q Rev Biol. 1982;57:29–52.CrossRef Kingsland S. The refractory model: the logistic curve and the history of population ecology. Q Rev Biol. 1982;57:29–52.CrossRef
28.
go back to reference Gottschalk PG, Dunn JR. The five-parameter logistic: a characterization and comparison with the four-parameter logistic. Anal Biochem. 2005;343:54–65.CrossRef Gottschalk PG, Dunn JR. The five-parameter logistic: a characterization and comparison with the four-parameter logistic. Anal Biochem. 2005;343:54–65.CrossRef
29.
go back to reference Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. 2011;342:d549.CrossRef Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. 2011;342:d549.CrossRef
30.
go back to reference Kriston L. Dealing with clinical heterogeneity in meta-analysis. Assumptions, methods, interpretation. Int J Meth Psych Res. 2013;22:1–15.CrossRef Kriston L. Dealing with clinical heterogeneity in meta-analysis. Assumptions, methods, interpretation. Int J Meth Psych Res. 2013;22:1–15.CrossRef
31.
go back to reference Lunn DJ, Thomas A, Best N, Spiegelhalter D. WinBUGS - a Bayesian modelling framework: concepts, structure, and extensibility. Stat Comput. 2000;10:325–37.CrossRef Lunn DJ, Thomas A, Best N, Spiegelhalter D. WinBUGS - a Bayesian modelling framework: concepts, structure, and extensibility. Stat Comput. 2000;10:325–37.CrossRef
32.
go back to reference Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420–8.CrossRef Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420–8.CrossRef
33.
go back to reference Diez R. A glossary for multilevel analysis. J Epidemiol Community Health. 2002;56:588–94.CrossRef Diez R. A glossary for multilevel analysis. J Epidemiol Community Health. 2002;56:588–94.CrossRef
34.
go back to reference Kriston L, Meister R. Incorporating uncertainty regarding applicability of evidence from meta-analyses into clinical decision making. J Clin Epidemiol. 2014;67:325–34.CrossRef Kriston L, Meister R. Incorporating uncertainty regarding applicability of evidence from meta-analyses into clinical decision making. J Clin Epidemiol. 2014;67:325–34.CrossRef
35.
go back to reference Kriston L. Aktuelle Entwicklung der kumulativen Inzidenz bestätigter SARS-CoV-2-Infektionen und infektionsbedingter Todesfälle in Deutschland. [Modeling the cumulative incidence of SARS-CoV-2 cases and deaths in Germany]. [German]. OSF Preprints. Published 5 May 2020. https://doi.org/10.31219/osf.io/q2yw5. Kriston L. Aktuelle Entwicklung der kumulativen Inzidenz bestätigter SARS-CoV-2-Infektionen und infektionsbedingter Todesfälle in Deutschland. [Modeling the cumulative incidence of SARS-CoV-2 cases and deaths in Germany]. [German]. OSF Preprints. Published 5 May 2020. https://​doi.​org/​10.​31219/​osf.​io/​q2yw5.
36.
go back to reference King AA. Domenech de Cellès M, Magpantay FMG, Rohani P. Avoidable errors in the modelling of outbreaks of emerging pathogens, with special reference to Ebola. Proc Biol Sci. 2015;282:20150347. King AA. Domenech de Cellès M, Magpantay FMG, Rohani P. Avoidable errors in the modelling of outbreaks of emerging pathogens, with special reference to Ebola. Proc Biol Sci. 2015;282:20150347.
37.
go back to reference Scarpino SV, Petri G. On the predictability of infectious disease outbreaks. Nat Commun. 2019;10:898.CrossRef Scarpino SV, Petri G. On the predictability of infectious disease outbreaks. Nat Commun. 2019;10:898.CrossRef
38.
go back to reference May RM. Uses and abuses of mathematics in biology. Science. 2004;303:790–3.CrossRef May RM. Uses and abuses of mathematics in biology. Science. 2004;303:790–3.CrossRef
39.
go back to reference Razum O, Becher H, Kapaun A, Junghanss T. SARS, lay epidemiology, and fear. Lancet. 2003;361:1739–40.CrossRef Razum O, Becher H, Kapaun A, Junghanss T. SARS, lay epidemiology, and fear. Lancet. 2003;361:1739–40.CrossRef
40.
go back to reference Jewell NP, Lewnard JA, Jewell BL. Caution warranted: using the Institute for Health Metrics and Evaluation Model for predicting the course of the COVID-19 pandemic. Ann Intern Med. 2020;173:226–7.CrossRef Jewell NP, Lewnard JA, Jewell BL. Caution warranted: using the Institute for Health Metrics and Evaluation Model for predicting the course of the COVID-19 pandemic. Ann Intern Med. 2020;173:226–7.CrossRef
41.
go back to reference Kriston L. Machine learning’s feet of clay. J Eval Clin Pract. 2020;26:373–5.CrossRef Kriston L. Machine learning’s feet of clay. J Eval Clin Pract. 2020;26:373–5.CrossRef
42.
go back to reference Moran KR, Fairchild G, Generous N, Hickmann K, Osthus D, Priedhorsky R, et al. Epidemic forecasting is messier than weather forecasting: the role of human behavior and internet data streams in epidemic forecast. J Infect Dis. 2016;214(Suppl 4):S404–8.CrossRef Moran KR, Fairchild G, Generous N, Hickmann K, Osthus D, Priedhorsky R, et al. Epidemic forecasting is messier than weather forecasting: the role of human behavior and internet data streams in epidemic forecast. J Infect Dis. 2016;214(Suppl 4):S404–8.CrossRef
Metadata
Title
Predictive accuracy of a hierarchical logistic model of cumulative SARS-CoV-2 case growth until May 2020
Author
Levente Kriston
Publication date
01-12-2020
Publisher
BioMed Central
Keyword
SARS-CoV-2
Published in
BMC Medical Research Methodology / Issue 1/2020
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-020-01160-2

Other articles of this Issue 1/2020

BMC Medical Research Methodology 1/2020 Go to the issue