Enhancing Transformer-Based Foundation Models for Time Series Forecasting via Bagging, Boosting and Statistical Ensembles
Dhruv D. Modi, Rong Pan · 2025
Read Enhancing Transformer-Based Foundation Models for Time Series Forecasting via Bagging, Boosting and Statistical Ensembles by Dhruv D. Modi, Rong Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16641.txt.
Introduction
Dhruv Modi and Rong Pan show in this 2025 paper that old statistical teamwork makes new forecasting giants steadier. Time series foundation models predict well out of the box, yet wobble with variance, local bias, and vague uncertainty on real data. On a Belgian electricity load set, the authors wrapped such models in bagging, stacking, residual modeling, and feedback loops.
Every hybrid beat the lone models across horizons. Stacked ensembles scored the lowest error, resampling tamed long-range misses, residual models scrubbed steady bias, and the built intervals covered truth near their promised rate. The authors argue statistics and foundation models gain from each other. Grid operators care most about those long-range misses.
Catalog record
| CREATOR | Dhruv D. Modi, Rong Pan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16641
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Transformer-Based Foundation Models for Time Series Forecasting via Bagging, Boosting and Statistical Ensembles Dhruv D. Modi, Rong Pan Abstract Time series foundation models (TSFMs) such as Lag-Llama, TimeGPT, Chronos, MOMENT, UniTS, and TimesFM have shown strong generalization and zero-shot capabilities for time series forecasting, anomaly detection, classification, and imputation. Despite these advantages, their predictions still suffer from variance, domain-specific bias, and limited uncertainty quantification when deployed on real operational data. This paper investigates a suite of statistical and ensemble-based enhancement techniques—bootstrap-based bagging, regression-based stacking, prediction-interval construction, statistical residual modeling, and iterative error feedback—to improve robustness and accuracy. Using the Belgium Electricity Short-Term Load Forecasting dataset as a case study, we demonstrate that the proposed hybrids consistently outperform standalone foundation models across multiple horizons. Regression-based ensembles achieve the lowest mean squared error; bootstrap aggregation markedly reduces long-context errors; residual modeling corrects systematic bias; and the resulting prediction intervals achieve near-nominal coverage with widths shrinking as context length increases. The results indicate that integrating statistical reasoning with modern foundation models yields measurable gains in accuracy, reliability, and interpretability for real-world time series applications. Keywords: time series forecasting; foundation models; Transformers; ensembling; bagging; boosting; uncertainty quantification 1 Introduction Accurate time series forecasting plays a central role in decision-making for energy, finance, healthcare, and retail. Traditional statistical models, including ARIMA/SARIMA and exponential smoothing, remain widely used because they are interpretable and computationally light ( Montgomery et al. 2015 ; Liang et al. 2024 ) . However, they often struggle with non-linear dependencies, regime changes, and long-range structure. Deep learning approaches based on CNNs and RNNs/LSTMs mitigate some of these issues, yet face limitations in capturing very long dependencies and often require heavy task-specific tuning. Transformer architectures address these issues with self-attention and scale favorably with data ( Jiang et al. 2024 ) . Recently, foundation models for time series have emerged, leveraging pretraining and adaptation to deliver zero-shot generalization and strong accuracy across tasks and domains ( Rasul et al. 2023 ; Garza and Mergenthaler-Canseco 2023 ; Ansari et al. 2024 ; Goswami et al. 2024 ; Gao et al. 2024 ; Woo et al. 2024 ) . Still, three obstacles remain persistent in practice: (i) high variance for single-model forecasts, (ii) domain-specific bias when pretrained models meet specialized seasonal or calendar effects, and (iii) insufficient uncertainty quantification when only point forecasts are provided. This work augments foundation models with statistical post-processing and hybridization. We propose: (1) bootstrap-based bagging to stabilize probabilistic forecasts; (2) regression-based ensembling that balances complementary inductive biases; (3) principled prediction intervals (PIs) for uncertainty communication; and (4) residual modeling and iterative error feedback for bias correction and online adaptation. We evaluate these methods on an operational electricity load forecasting task and show consistent improvements over foundation-model baselines. This paper represents a condensed version of Dhruv Modi’s master’s thesis, Foundation Models for Time Series , completed at Arizona State University in 2025 ( Modi 2025 ) . Contributions. We provide a unified methodology for integrating bagging, stacking, prediction intervals (PIs), and residual boosting with Transformer-based foundation models; present empirical evidence that these hybrids improve 1–24 hour forecasting accuracy and reduce variance on a real-world load dataset; and deliver a PI construction for ensembles that achieves near-nominal coverage while narrowing with longer historical context. The rest of this paper is organized as follows: Section 2 reviews related work in statistical and foundation model-based time series forecasting. Section 3 details the proposed methodology, including bagging, boosting, and ensembling strategies. Section 4 describes the experimental setup. Section 5 presents results and discussion, and Section 6 concludes with implications and directions for future research. 2 Related Work 2.1 Classical Forecasting Autoregressive and moving-average models, exponential smoothing, and their seasonal variants remain cornerstones of industrial forecasting ( Montgomery et al. 2015 ; Liang et al. 2024 ) . Their strengths include interpretability and strong performance on stationary or near-stationary processes, yet they can underfit complex nonlinear structure and long-range dependencies. 2.2 Deep Learning for Time Series CNNs and RNNs/LSTMs have been adapted to forecasting, often improving non-linear modeling. Nevertheless, they may suffer from vanishing gradients, limited receptive fields, and heavy hyperparameter tuning for long-context tasks. 2.3 Transformer-Based Foundation Models Self-attention enables modeling of long-range dependencies without recurrence ( Jiang et al. 2024 ) . Several time-series foundation models (TSFMs) extend this paradigm: Lag-Llama is a decoder-only univariate forecaster using RMSNorm and RoPE with a distribution head for probabilistic outputs ( Rasul et al. 2023 ; Liang et al. 2024 ; Liang et al. 2024 ) . TimeGPT adds multi-horizon zero-shot forecasting with exogenous variables in an encoder–decoder design ( Garza and Mergenthaler-Canseco 2023 ) . Chronos treats time series as tokens via scaling and quantization to repurpose T5-like language models for forecasting and quantiles ( Ansari et al. 2024 ; Jiang et al. 2024 ) . MOMENT uses masked time-series modeling with an encoder-only backbone and patching ( Goswami et al. 2024 ) . UniTS introduces unified multitask and multidomain modeling via task tokenization and dynamic attention ( Gao et al. 2024 ) . TimesFM employs patch-based tokenization and decoder-only transformers trained on a mixture of real and synthetic data for long-horizon forecasting ( Woo et al. 2024 ) . While these works demonstrate strong generalization, systematic study of statistical hybridization (bagging, stacking, residual correction) for TSFMs remains limited. 3 Methodology We integrate bootstrap-based bagging, regression-based stacking, prediction-interval construction, residual modeling, and iterative error feedback into forecasting pipelines built on foundation models. The methods target variance reduction, bias correction, and uncertainty reporting. We use AutoGluon ( Shchur et al. 2023 ) , which is an open-source AutoML toolkit from AWS, to automate statistical/machine-learning model selection, ensembling, and hyperparameter tuning. 3.1 Bagging: Bootstrap-Based Enhancement For each time step, the base foundation model (Lag-Llama) produces a point forecast and a sample of n{=}100 one-step-ahead draws \{\hat{y}^{(i)}_{t}\}_{i=1}^{n} . We draw b{=}40 values with replacement to compute a bootstrap mean \bar{y}^{(j)}_{t}=\frac{1}{b}\sum_{i=1}^{b}\hat{y}^{(i)}_{t} , repeat this m{=}100 times, and average to obtain the final forecast \hat{y}^{\text{bag}}_{t}=\frac{1}{m}\sum_{j=1}^{m}\bar{y}^{(j)}_{t} . This aggregation reduces variance relative to single-sample predictions. Algorithm 1 Bootstrap-Enhanced Lag-Llama Forecasting 1: Input : Univariate time series 2: Output : Prediction from Point Prediction Model and the Bagging model along with their MSE. 3: Step a: Lag-Llama Prediction Process: 4: Select past {1 week, 3 weeks, 5 weeks} as context window. 5: Perform zero shot forecasting to obtain Point Prediction and sample of 100 values for next hour (one-step-ahead) forecast. 6: The context window slides by 1 hour, and the process repeats. 7: Step b: Bootstrap Mean Calculation: 8: Sample 40 values from the 100 values (with replacement) and compute the mean. 9: Step c: Generate 100 Bootstrap Mean Columns: 10: Apply Step b to the prediction sample 100 times. 11: Step d: Compute Mean: 12: bootstrap_mean_all: Mean of all bootstrap mean columns generated in Step c. 13: Step e: Evaluation: 14: Compute MSE between: • Point Prediction and actual. • Compute MSE between bootstrap_mean_all and actual. Algorithm 1 : Bootstrap-Enhanced Lag-Llama Forecasting improves the accuracy of univariate time series predictions by integrating statistical bagging techniques with the transformer-based Lag-Llama model. The process begins with Step a, where prediction from Lag-llama is obtained. Here, model selects a contextual window of historical data