History

Reich, N. G., Lessler, J., Funk, S., Viboud, C., Vespignani, A., Tibshirani, R. J., … & Biggerstaff, M. (2022). Collaborative hubs: making the most of predictive epidemic modeling. American journal of public health, 112(6), 839-842.

Reference

A thorough discussion on forecasting hubs for infectious diseases, including the reasoning behind them can be found in this short paper on collaborative hubs.

In the early 2010s there were many infectious disease forecasting models, but most were published months to years after the forecasted event. These models often had very different targets they were forecasting, such as weekly incidence, epidemic duration, monthly visits, and time of peak incidence. Additionally, the accuracy of these models were evaluated using very different metrics, such as the mean absolute error, median absolute error, and correlation. All of these factors made comparing these forecast models in a fair, consistent, and accurate way, very challenging. In a scoping review of influenza forecasting in human populations published in 2014, Chretien et al concluded that:

Chretien, J. P., George, D., Shaman, J., Chitale, R. A., & McKenzie, F. E. (2014). Influenza forecasting in human populations: a scoping review. PloS one, 9(4), e94130.

“Comparing the accuracy of forecasting applications is difficult because forecasting methods, forecast outcomes, and reported validation metrics varied widely.”

At the same time, there were various efforts to aggregate and evaluate forecasts for infectious diseases in order to better address the challenges they pose from a public health perspective. Below is a timeline of some of the major “hubs” that have been developed since 2013.

Timeline showing various major infectious disease forecasting “hubs”

Figure credits: Alex Vespignani and Nicole Samay.

The Hubverse is an effort at providing tools and a unified framework for aggregating, visualizing, and evaluating forecasts. Infectious disease forecasting was the motivation for developing these tools, but the approach is meant to be versatile so that the Hubverse can be used for other types of forecasts.

Motivation

There are many reasons to support collaborative modelling, rather than relying on a single forecasting model. Firstly, ensemble models are both more accurate and useful than individual models:

• A simple average across models has better predictive performance

• Ensemble models are especially reliable for predicting across multiple targets

Additionally, there are scientific and structural benefits to modelling via Hubs:

• They allow comparable evaluation across different models

• They provide opportunities for scientific exchange, and for method & data sharing

• Hubs have a greater transparency of model outputs

• Hubs can also be a venue for communication with stakeholders

In working on collaborative forecasting, our major objectives have been to

• Connect forecasting research to decision making needs

• Evaluate forecast skill and facilitate forecasting research

• Operationalize forecasting

• Ensure increased data availability and real-time forecasting

• Forecast and evaluation standardization

• Community building (e.g., CSTE, academia, industry)

• Aggregate and coordinate the work done by varied forecasting teams