The cost of ensembling: is it always worth combining?

Marco Zanotti · 2025

Read The cost of ensembling: is it always worth combining? by Marco Zanotti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04677.txt.

Introduction

Marco Zanotti asks whether combining forecast models is always worth its computing bill. Forecast ensembles usually beat single models, above all for forecasts that carry uncertainty ranges. But bigger data and heavier models make each run cost more. Zanotti weighs the gains against the cost across two huge retail data sets.

Ten single models and eight blends were tested under different retraining schedules. Blends did win on accuracy, with the largest, accuracy-hungry blends costing the most. Retraining less often cut costs sharply while barely denting accuracy. Lean blends built for efficiency nearly matched the best blends at far lower cost. Most strikingly, blends of just two or three models often came near the top. The study offers plain rules for forecast teams that want accuracy without waste.

Catalog record

CREATORMarco Zanotti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04677

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The cost of ensembling: is it always worth combining? Marco Zanotti Abstract Given the continuous increase in dataset sizes and the complexity of forecasting models, the trade-off between forecast accuracy and computational cost is emerging as an extremely relevant topic, especially in the context of ensemble learning for time series forecasting. To asses it, we evaluated ten base models and eight ensemble configurations across two large-scale retail datasets (M5 and VN1), considering both point and probabilistic accuracy under varying retraining frequencies. We showed that ensembles consistently improve forecasting performance, particularly in probabilistic settings. However, these gains come at a substantial computational cost, especially for larger, accuracy-driven ensembles. We found that reducing retraining frequency significantly lowers costs, with minimal impact on accuracy, particularly for point forecasts. Moreover, efficiency-driven ensembles offer a strong balance, achieving competitive accuracy with considerably lower costs compared to accuracy-optimized combinations. Most importantly, small ensembles of two or three models are often sufficient to achieve near-optimal results. These findings provide practical guidelines for deploying scalable and cost-efficient forecasting systems, supporting the broader goals of sustainable AI in forecasting. Overall, this work shows that careful ensemble design and retraining strategy selection can yield accurate, robust, and cost-effective forecasts suitable for real-world applications. 1 Introduction Ensemble learning has become a powerful and widely adopted approach in time series forecasting due to its ability to improve predictive accuracy and robustness by aggregating the outputs of multiple models. The fundamental idea behind ensemble methods is that combining forecasts from diverse models can help balance their individual strengths and weaknesses, thereby reducing the risk of overfitting and mitigating model-specific biases or errors. This approach leverages the principle that while individual models might make different types of errors, their combination can average out these errors. One of the most intriguing and empirically validated findings in the literature is that simple ensemble strategies, such as taking the mean or median of forecasts, often outperform more sophisticated combination methods like stacking (also known as meta-learning) or weighted ensembles. This phenomenon, known as the forecast combination puzzle ( Claeskens et al. 2016 ) , suggests that in many cases the added complexity of estimating optimal weights or training a meta-learner does not yield meaningful gains in accuracy and can even lead to overfitting, especially when the number of forecast points is limited. Indeed, the success of simple combinations is partly due to their ability to remain robust in the presence of model misspecification, parameter uncertainty, and changing data dynamics, issues that more complex ensemble methods may struggle to accommodate effectively. In general, the benefits of ensemble learning are numerous: it improves forecast accuracy, enhances model stability across time and datasets, and increases resilience to structural breaks or shifts in the data. Ensembles also provide a practical hedge against model uncertainty, allowing practitioners to rely less on the assumption that any single model is correctly specified. In the context of global forecasting models, ensemble methods become especially valuable. Global models are trained across many time series simultaneously, and while they excel at identifying cross-series patterns, they can also suffer from model instability or biases if the chosen architecture fails to generalize across heterogeneous series. Combining multiple global models, each with different biases, architectures, or feature representations, can help counteract these issues by smoothing out individual model errors and reducing the risk of overfitting to shared but spurious signals across series ( Wang et al. 2023 ) . Additionally, ensembles serve as a hedge against model selection uncertainty, which is particularly relevant in global settings where the optimal model may vary substantially across groups of series. By pooling predictions, ensembles offer a form of model diversification that improves generalization, making them a powerful and practical tool in modern forecasting systems. However, ensemble methods are not without drawbacks. One significant limitation is the increased computational cost, especially when combining large numbers of complex models like deep learning architectures or global forecasting models. This cost is magnified when retraining or model tuning is required for each component model at multiple forecast origins. Additionally, from a business perspective, managing and maintaining multiple models adds complexity to the production of forecasts. Moreover, in cases where the component models are very similar or highly correlated, the marginal benefit of ensembling diminishes, potentially offering little improvement over the best individual model ( Kolassa 2011 ) . Therefore, one of the major advantages of adopting a global modeling approach can be offset when ensemble learning is introduced. Combining several models, even with simple techniques like averaging, inherently increases the computational time, as multiple models must be trained, validated, and maintained, producing substantial computational overhead. In this sense, ensemble learning may weaken one of the core benefits of the global modeling paradigm: reduced training time and resource usage. Moreover, since forecasts are typically generated using cloud computing services with pay-as-you-go pricing models ( Fotios Petropoulos & Spiliotis 2024 ) , increased computational time and resource consumption directly leads to higher forecasting costs for organizations. These costs may also rise exponentially when frequent retraining strategies are adopted. Indeed, it is a widespread practice to update forecasting models whenever new data becomes available, often driven by the belief that frequent updates enhance the model’s ability to adapt to evolving patterns and improve predictive accuracy. However, when ensembles are used, all the models within the ensembles must be retrained to obtain the final predictions, abruptly increasing the costs of forecasting. This trade-off between performance and computational time in the context of ensemble learning raises important questions about the balance between accuracy gains and cost efficiency, especially in large-scale forecasting applications. However, despite these challenges, ensemble learning remains one of the most widely used strategies in the forecasting industry, which further amplifies concerns about the long-term sustainability and scalability of forecasting systems. Indeed, from a practical perspective, forecast combinations, coupled with frequent retraining, has also significant environmental costs, since the energy consumption of model training extends beyond direct computational costs, contributing to energy consumption and carbon emissions ( Schwartz et al. 2020 ) . Therefore, understanding the cost of ensemble forecasting models and the effects of retraining on their performance is of paramount importance for advancing more sustainable forecasting practices. 1.1 Research Question We aim to address the question "Are ensembles always the solution?" . Specifically, we focus on the trade-off between accuracy and sustainability, in terms of the cost of producing the forecasts, among different types of ensembles of global forecasting models. This cost is particularly relevant for ensemble models since it becomes exponentially larger with the number of base models used to produce the combined forecasts. To this end, we use ten global models as base learners, ranging from traditional machine learning methods to advanced deep learning architectures. In addition, we evaluate eight ensemble approaches, all trained and tested on the two most recent and comprehensive retail forecasting datasets: the M5 and VN1 competition datasets. We also investigated the effects of retraining on the forecasting performance of ensembles, compared to that of the base models. In particular, we assess whether reducing the frequency of retraining, by avoiding re-estimation of base models as new data becomes available, can effectively enhance the cost-accuracy trade-off in ensemble forecasting. Hence, we explored a

Read the complete text