Temporal horizons in forecasting: a performance-learnability trade-off

Pau Vilimelis Aceituno, Jack William Miller, Noah Marti, Youssef Farag, Victor Boussange · 2025

Read Temporal horizons in forecasting: a performance-learnability trade-off by Pau Vilimelis Aceituno, Jack William Miller, Noah Marti, Youssef Farag, Victor Boussange (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03889.txt.

Introduction

Pau Vilimelis Aceituno and four co-authors study how far ahead a forecasting model should be trained to look, in a 2025 paper. Models that predict step by step are common for moving systems. Training them on longer horizons may help or hurt. The team links the shape of the training error surface to the chosen horizon.

They prove that error dips found for long horizons also serve short forecasts well. Dips found on short horizons, by contrast, give worse long forecasts. Yet the error surface grows rougher as the horizon lengthens. That roughness makes long-horizon training hard in practice. Tests on the computer backed the theory. The results give a principled base for picking training horizons. The authors discuss what this means for tuning such models in real work.

Catalog record

CREATORPau Vilimelis Aceituno, Jack William Miller, Noah Marti, Youssef Farag, Victor Boussange
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03889

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Temporal horizons in forecasting: a performance-learnability trade-off Pau Vilimelis Aceituno, Jack William Miller, Noah Marti, Youssef Farag, Victor Boussange Abstract When training autoregressive models to forecast dynamical systems, a critical question arises: how far into the future should the model be trained to predict? Too short a horizon may miss long-term trends, while too long a horizon can impede convergence due to accumulating prediction errors. In this work, we formalize this trade-off by analyzing how the geometry of the loss landscape depends on the training horizon. We prove that for chaotic systems, the loss landscape’s roughness grows exponentially with the training horizon, while for limit cycles, it grows linearly, making long-horizon training inherently challenging. However, we also show that models trained on long horizons generalize well to short-term forecasts, whereas those trained on short horizons suffer exponentially (resp. linearly) worse long-term predictions in chaotic (resp. periodic) systems. We validate our theory through numerical experiments and discuss practical implications for selecting training horizons. Our results provide a principled foundation for hyperparameter optimization in autoregressive forecasting models. 1 Introduction Forecasting the future state of dynamical systems is a fundamental challenge in scientific and engineering disciplines, from climate modeling to robotics. A central objective is to generate accurate predictions as far into the future as possible. Auto-regressive (AR) models, which iteratively feed their own predictions back as inputs, are a standard tool for this task in both traditional scientific computing and machine learning [ 56 , 40 , 5 , 67 , 41 ] . Traditional AR models were linear and limited to low-dimensional time series [ 56 , 40 ] . Modern auto-regressive neural networks (ARNNs), trained via gradient descent, have emerged as powerful alternatives due to their ability to capture nonlinear dynamics in high-dimensional systems. Such nonlinearity is ubiquitous across scientific domains, including physics [ 42 ] , robotics [ 33 ] , ecology [ 7 , 32 ] , and climate science [ 30 ] . Consequently, ARNNs have demonstrated success in climate prediction [ 41 , 38 ] , physics-informed dynamical system emulation [ 5 , 17 , 43 , 51 ] , and robotic control [ 63 , 70 , 25 ] . Notably, these systems are typically Markovian, meaning their future states depend only on the current state. This property renders architectures with explicit memory mechanisms, such as recurrent neural networks (RNNs) or transformers, unnecessarily complex and prone to overfitting. A critical yet understudied challenge in training ARNNs is selecting an appropriate temporal horizon for prediction. Common approaches either use a single-step prediction scheme [ 67 , 5 , 51 ] or adopt longer horizons without rigorous justification [ 41 , 38 , 59 ] . Currently, no theoretical framework exists to guide this choice. This gap extends to mechanistic modeling with differential equations, where methods like piecewise regression [ 57 , 14 , 10 ] and multiple shooting [ 9 , 16 , 4 ] similarly lack principled criteria for horizon selection. In this work, we rigorously analyze how the training temporal horizon affects ARNN performance. We derive practical guidelines for selecting this horizon based on the underlying system dynamics. Our contributions include: • A theoretical analysis linking the training horizon to the geometry of the loss landscape • Empirical validation of these relationships across different dynamical systems • Practical recommendations for horizon selection in both data-driven and mechanistic modeling The remainder of this paper is organized as follows. We first situate our work within the existing literature. Next, we introduce necessary background and notation. We then present our theoretical analysis of how temporal horizons shape the loss landscape, that we illustrate with numerical experiments. we examine the practical implications of our theoretical findings under the presence of noise and various model sizes, and how it connects with the task of fitting mechanistic models. Finally, we discuss broader implications and future research directions. 2 Related works While ARNNs are a natural choice for forecasting Markovian systems, much of the literature on general time series forecasting has focused on recurrent neural networks (RNNs) [ 22 ] . RNNs, such as LSTMs [ 27 ] , leverage recurrent connections to maintain memory of past inputs, enabling them to capture temporal dependencies. More recently, transformers [ 68 ] have emerged as a competitive alternative, often outperforming RNNs in practice [ 19 ] . Training RNNs or transformers requires backpropagation through time, where gradients are computed across sequences whose length corresponds to the temporal horizon. However, this process is often hampered by the vanishing or exploding gradient problem (EVGP) [ 55 , 29 , 6 ] . Common strategies to mitigate EVGP include architectural modifications (e.g., gated mechanisms [ 27 ] ), specialized weight initializations [ 52 ] , gradient clipping [ 21 , 73 ] , and constrained dynamics (e.g., unitary RNNs [ 3 , 12 , 18 ] ). Despite these advances, RNNs often underperform for dynamical systems forecasting, as stability-focused designs can limit expressivity [ 54 , 64 , 50 ] . Recent work has proposed dynamically adjusting the BPTT horizon based on system properties like Lyapunov exponents [ 50 ] , though such approaches still lag behind ARNNs in practice. In contrast to RNNs, ARNNs lack a robust theoretical framework connecting their training horizon to the underlying system dynamics. Some works adapt the horizon during training to balance stability and accuracy [ 10 , 57 ] , while others implicitly control it via heuristic strategies. For example, in weather forecasting [ 41 ] and financial time series modeling [ 8 ] , discount factors reduce the influence of distant prediction errors, effectively shortening the training horizon. Similarly, neural PDE solvers often prioritize high-frequency components, which are inherently tied to shorter temporal dependencies [ 69 , 44 , 39 ] . Interestingly, the choice of training horizon extends beyond forecasting. In reinforcement learning, the effective horizon influences model success, leading to techniques like \lambda -returns [ 65 , 2 ] . Model predictive control also requires careful horizon selection [ 26 ] . Even in language modeling, recent empirical results suggest that multi-token prediction improves performance [ 20 , 13 ] , though the theoretical basis remains unclear. 3 Background and Notation 3.1 Dynamical systems Dynamical systems can generally be described by a set of update equations of the form x(t)=\phi(x(t-\Delta t)) (1) where x(t)\in\mathcal{X} is the state of the system and \phi is a deterministic map and \Delta t is the timestep size, which in our derivations will be set to one for simplicity. Dynamical systems theory categorizes systems in four main categories: stable, unstable, limit cycles, and chaotic systems [ 34 , 11 ] . Stable systems are those that converge to an attractor state x^{*} , and unstable those that diverge from a given point x^{*} , and in both cases the point x^{*} defines a constant orbit \phi(x^{*})=0 . Those are well understood dynamics that are either inherently easy to predict – for stable systems, as the trajectories end in the same point – , or inherently impossible – for unstable systems, as the trajectories are constantly in a new state, and thus out of distribution. The more interesting families for machine learning

Read the complete text