Time Series Embedding and Combination of Forecasts: A Reinforcement Learning Approach
Marcelo C. Medeiros, Jeronymo M. Pinto · 2025
Read Time Series Embedding and Combination of Forecasts: A Reinforcement Learning Approach by Marcelo C. Medeiros, Jeronymo M. Pinto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20795.txt.
Introduction
Marcelo Medeiros and a co-author tackle a known puzzle in forecasting, in a paper published in 2025. Averaging forecasts from many methods is hard to beat, even with smart weighting schemes. Their answer is to let a learning agent pick and mix forecasts as conditions change, rather than fixing the mix in advance.
The framework treats each choice of mix as an action and learns from how well it pays off. The team tested it on made-up data and on two real sets: a large public contest archive and a survey of expert forecasters. Results support the method as a flexible way to combine forecasts under doubt. The authors offer it as a step toward settling the old puzzle of why simple averages are so hard to outdo.
Catalog record
| CREATOR | Marcelo C. Medeiros, Jeronymo M. Pinto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-20795
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Time Series Embedding and Combination of Forecasts: A Reinforcement Learning Approach Marcelo C. Medeiros, Jeronymo M. Pinto Abstract The forecasting combination puzzle is a well-known phenomenon in forecasting literature, stressing the challenge of outperforming the simple average when aggregating forecasts from diverse methods. This study proposes a Reinforcement Learning - based framework as a dynamic model selection approach to address this puzzle. Our framework is evaluated through extensive forecasting exercises using simulated and real data. Specifically, we analyze the M4 Competition dataset and the Survey of Professional Forecasters (SPF). This research introduces an adaptable methodology for selecting and combining forecasts under uncertainty, offering a promising advancement in resolving the forecasting combination puzzle. Keywords: Reinforcement Learning, Forecasting, Model Selection, Combination Puzzle. JEL codes : C32, C41, C44, C45, C50, C53 1 Introduction The study of forecast combinations has persistently captured considerable interest within the forecasting literature, particularly in Economics. A pivotal contribution was made by Bates and Granger 1969 , who explored the characteristics of forecast combinations and assessed whether integrating forecasts from multiple models could surpass the performance of simple averages. In an early review, Clemen 1989 concluded that equally weighting individual forecasts frequently constitutes the most efficacious approach. This conclusion led to the emergence of the “ forecasting combination puzzle ”. Despite extensive empirical findings and numerous simulation studies, optimal forecast combination strategies often fail to surpass the simple average. This conundrum underscores a critical issue: no single model consistently prevails over others in all situations. Only if a universal “winner” could be determined for each scenario would a superior alternative to simple averaging emerge. Suppose that at a given moment t , and utilizing the available information set, an agent intends to construct a forecast \widehat{Y}_{t+h|t} for a target variable Y_{t+h} , where h>0 denotes the forecast horizon. Furthermore, let us assume that at time t , the agent possesses n possible forecasts derived from various methods or models, \widehat{Y}_{t+h|t}^{(a)} , a=,\ldots,n . Among these available models or methods, it is presumed that several canonical model combinations are incorporated. For instance, a simple average of forecasts might exemplify one such model. Consequently, at each period, the agent (forecaster) will take an action by selecting a forecast \widehat{Y}_{t+h|t}^{(a_{0})} , a_{0}\in\{1,\ldots,n\} , based on a chosen criterion function. A common strategy would be to opt for the model with the lowest mean squared error (MSE) over a recent historical span, such as the last \tau time intervals. Nevertheless, this approach may not prove optimal in contexts characterized by instability or structural breaks. Therefore, in this paper, we introduce an algorithm grounded in Reinforcement Learning (RL) designed to identify the most suitable model under analogous environmental conditions observed in antecedent periods. Our approach is loosely motivated by Franklin et al. 2024 , which presented a novel semantic search tool to identify historical news articles most similar to contemporary news queries. Feng and Zhang 2019 and Pinto and Castle 2022 provided evidence of the potential of Reinforcement Learning (RL) for model selection and its adaptability to structural changes. However, no research definitively establishes RL as the optimal framework for forecasting in scenarios where the data-generating process (DGP) is unknown. In this paper, we propose a novel solution utilizing RL to dynamically and adaptively select the optimal weights, thereby enhancing forecasting accuracy. We validate our approach through a comprehensive set of experiments on simulated data as well as two real-world datasets: the M4 Competition dataset and the Survey of Professional Forecasters (SPF) dataset. 1 1 1 https://forecasters.org/resources/time-series-data/ 2 2 2 https://www.philadelphiafed.org/ 2 Reinforcement Learning As described by Elavarasan and Vincent 2020 , RL is a dynamic programming framework that rewards successful decisions. Unlike traditional Machine Learning (ML) approaches, RL agents learn through interaction with their environment and focus on optimizing actions based on cumulative rewards. This framework has garnered significant attention within the ML community; see, for example, Ji et al. 2020 and Dong et al. 2020 . Reinforcement Learning (RL) represents a Sequential Decision Problem (SDP), characterized by the following essential components. • State Variable ( S_{t} ): Encodes relevant information for agent’s decision-making. • Decision Variables ( A_{t} ): Actions taken by the agent, guided by a policy rule ( \pi ). • Exogenous Information ( \boldsymbol{W}_{t+1} ): External factors (variables). • Transition Function: Updates the state based on decisions and external information: S_{t+1}=S^{M}(S_{t},A_{t},\boldsymbol{W}_{t+1}), where S^{M} is an arbitrary function that maps the previous state into the current state. The expected rewards for each state-action pair ( S_{t},A_{t} ) are updated based on the environment’s transition function and are continuously stored in the so-called Q-table , Q(S_{t},A_{t}) . • Objective Function: The goal is to maximize cumulative rewards or minimize penalties, R_{t} : v_{\pi}(S_{t})=\mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1}\middle|S_{t}=s\right], where \gamma is the discount factor. Among reinforcement learning (RL) methods, Temporal Difference (TD) learning is widely used due to its ability to update estimates directly from experience, without requiring a model of the environment or waiting for the end of an episode — a crucial advantage in non-stationary settings ( Sutton 1988 ) . TD learning forms the foundation of value-based methods such as Q-learning , which maintains a Q-table containing estimates of the expected return for each state-action pair. The Q-table is updated incrementally using the Bellman equation: Q(S_{t},A_{t})\leftarrow Q(S_{t},A_{t})+\alpha\left[R_{t+1}+\gamma\max_{a}Q(S_{t+1},a)-Q(S_{t},A_{t})\right]. (1) In this formulation, Q(S_{t},A_{t}) is the current reward for taking action A_{t} in state S_{t} , \alpha is the learning rate ( 0<\alpha\leq 1 ), \gamma is the discount factor ( 0\leq\gamma<1 ), R_{t+1} is the reward observed after the action, and S_{t+1} is the next state. The term in brackets is the TD error , measuring the difference between the predicted and actual reward. The Q-value is adjusted in the direction that minimizes this error. In our forecasting framework, we adapt this principle by interpreting each state S_{t} as a snapshot of the cumulative Mean Squared Errors (MSEs) up to time t , and each action A_{t} as the selection of a forecasting model. The reward R_{t+1} corresponds to the negative forecasting error at time t+1 , since our objective is to minimize error rather than maximize reward. This update enables the RL agent to learn which forecasting models perform best under varying historical error profiles, progressively enhancing model selection as more data becomes available. The goal is to develop an algorithm capable of retaining past data patterns and selecting the most appropriate model for each scenario. In this sense, the algorithm is designed to behave similarly to a researcher who, when faced with economic data, must determine the most suitable forecasting model for a given economic context. 3 Research Design 3.1 State Embedding Consider that at time t , the forecaster assembles a p\times t matrix \boldsymbol{\mathcal{E}}_{t} comprising features of each model from the n models/methods available, in conjunction with environmental factors. For instance, suppose the target variable is the next-period inflation. In this case, \boldsymbol{\mathcal{E}}_{t} may include the squared errors for each period in time, the cumulative forecast errors, several other performance metrics of each model or method (mean squared or absolute errors, for instance), as well as macroeconomic variables to reflect the current state of the economy, word counts from news data, among many other features. It is noteworthy that