Examining the Challenges of COVID-19 Modeling

Predicting the course of COVID-19 has been difficult.  Modeling the disease has as well.  The modeling experts at FiveThirtyEight eloquently described the challenges of modeling the disease in their article, “Why It’s So Freaking Hard to Make a Good COVID-19 Model.”

With the Coronavirus, there are so many disease parameters and social behavior assumptions that we don’t completely understand – even after ten months of the pandemic—it tests normal epidemiological modeling.  Throughout the pandemic, COVID-19 predictive models have come under criticism for conflicting predictions that have caused some to believe that all models are wrong; however, some models are useful.  For public health officials and decision-makers, it is helpful to know the expected short-term behavior of the COVID-19 pandemic.

Despite the modeling challenges, many entities willing entered the foray of modeling COVID-19 and projecting future new cases.  The Centers for Disease Control and Prevention (CDC) receives forecasts of new COVID-19 cases from modeling groups and presents those projections as an ensemble forecast.  As of November 2, 2020, the CDC website states, “Over the last several weeks, more reported cases than expected have fallen outside of the forecasted prediction intervals. This suggests that current forecast prediction intervals may not reflect the full range of future reported case numbers. Forecasts for new cases should be interpreted accordingly.”  In other words, the actual number of COVID-19 cases falls outside their 95% prediction intervals, and one should use caution when interpreting the ensemble forecast.

Figure 1 below shows when the problem likely happened.

CDC’s National COVID-19 Forecast
Figure 1. CDC’s National COVID-19 Forecast (Source: https://www.cdc.gov/coronavirus/2019-ncov/covid-data/forecasting-us-cases-previous.html)

After seven weeks of steady declines in the number of new COVID-19 cases per week, there was an uptick observed on September 21, 2020 (left graph in Figure 1).  This uptick in new cases is a challenge for epidemiological models as the models try to determine whether the increase is an anomaly, a transient increase, or the beginning of a new increase.  The CDC ensemble (indicated by the red dots and line) predicts that the number of new cases per week will decline over the next few weeks.  At this same time, IEM’s AI COVID model predicted that the increase in cases was real and projected that new cases in the next three weeks would grow by 133% (see Table 1).  After another week of COVID-19 cases, the IEM AI model projected that in three weeks (October 17), there would be 388,045 new cases for that week, with a growth rate of 126% (actual cases observed was 384,793 and a growth rate of 129%).  The IEM projections contrast with the CDC ensemble forecast, which remains relatively constant over the next few weeks, with a growth rate of 101%-103%.

Comparing IEM’s AI Model, CDC Ensemble Model, and actual COVID-19 case numbers
Table 1.  Comparing IEM’s AI Model, CDC Ensemble Model, and actual COVID-19 case numbers.

The accuracy in a three-week projection has higher uncertainty than shorter, one-week projections, which utilize more recent data to improve forecasted projections.  This is demonstrated in comparing the IEM AI-based model with that of the CDC ensemble forecast.  On October 31, there were 551,194 new confirmed COVID-19 cases for the week.  Looking at the data, the CDC ensemble model predicted 388,838 new cases while the IEM AI model projected 505,441 new cases for that week.

Even after five straight weeks of increasing new weekly COVID-19 cases (right graph in Figure 1), the CDC ensemble model is still projecting an almost constant growth rate.  This highlights a caveat for the CDC ensemble, as it attempts to “average out” all the predictions from the modeling groups.  You can see in the two graphs that the individual models range from decreasing projections to increasing projections, which ultimately leads to an ensemble COVID case projection that is almost constant.

While we acknowledge the challenges involved in modeling COVID-19, consumers of models need to understand how a specific model or an ensemble of models works to better understand how to interpret, and thus use, the results.

Learn more about IEM’s AI COVID-19 model or contact one of our experts.

Insights by

Sid Baccam, PhD
Manager, Emerging Technologies

Sources:

Johns Hopkins COVID Dashboard

CDC COVID-19 Forecasts