Understanding the Implicit Biases of Design Choices for Time Series Foundation Models
Annan Yu, 1, Danielle C. Maddix, 2, Boran Han, 2 Xiyuan Zhang, 2 Abdul Fatir Ansari, 2 Oleksandr Shchur, 2 Christos Faloutsos, 3 Andrew Gordon Wilson, 4 Michael W. Mahoney, 4 Yuyang Wang 2 1 Center for Applied Mathematics, Cornell University, Ithaca, NY 14853, USA 2 Amazon Web Services (3075 Olcott · 2025
Read Understanding the Implicit Biases of Design Choices for Time Series Foundation Models by Annan Yu, 1, Danielle C. Maddix, 2, Boran Han, 2 Xiyuan Zhang, 2 Abdul Fatir Ansari, 2 Oleksandr Shchur, 2 Christos Faloutsos, 3 Andrew Gordon Wilson, 4 Michael W. Mahoney, 4 Yuyang Wang 2 1 Center for Applied Mathematics, Cornell University, Ithaca, NY 14853, USA 2 Amazon Web Services (3075 Olcott (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19236.txt.
Introduction
Annan Yu and nine co-authors examine how quiet design choices bias time series foundation models, rather than chasing another leaderboard win. These general-purpose forecasting models are shaped by details such as patch size, embedding choice, and training objective. Each choice tilts what the model learns about time.
Through theory and controlled tests, the authors show how those knobs change temporal behavior, geometric structure, and how strongly a model snaps back toward average values. Understanding such hidden biases helps users pick and audit models for forecasting tasks where a subtle tilt can quietly distort every prediction the system makes.
Catalog record
| CREATOR | Annan Yu, 1, Danielle C. Maddix, 2, Boran Han, 2 Xiyuan Zhang, 2 Abdul Fatir Ansari, 2 Oleksandr Shchur, 2 Christos Faloutsos, 3 Andrew Gordon Wilson, 4 Michael W. Mahoney, 4 Yuyang Wang 2 1 Center for Applied Mathematics, Cornell University, Ithaca, NY 14853, USA 2 Amazon Web Services (3075 Olcott |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 130 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19236
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding the Implicit Biases of Design Choices for Time Series Foundation Models Annan Yu, 1, Danielle C. Maddix, 2, Boran Han, 2 Xiyuan Zhang, 2 Abdul Fatir Ansari, 2 Oleksandr Shchur, 2 Christos Faloutsos, 3 Andrew Gordon Wilson, 4 Michael W. Mahoney, 4 Yuyang Wang 2 1 Center for Applied Mathematics, Cornell University, Ithaca, NY 14853, USA 2 Amazon Web Services (3075 Olcott St., Santa Clara, CA 95054, USA) 3 Amazon Selling Partner Services (501 Fairview Ave. N., Seattle, WA 98109, USA) 4 Amazon Supply Chain Optimization Technologies (7 West 34th St., New York, NY 10001, USA) Abstract Time series foundation models (TSFMs) are a class of potentially powerful, general-purpose tools for time series forecasting and related temporal tasks, but their behavior is strongly shaped by subtle inductive biases in their design. Rather than developing a new model and claiming that it is better than existing TSFMs, e.g., by winning on existing well-established benchmarks, our objective is to understand how the various “knobs” of the training process affect model quality. Using a mix of theory and controlled empirical evaluation, we identify several design choices (patch size, embedding choice, training objective, etc.) and show how they lead to implicit biases in fundamental model properties (temporal behavior, geometric structure, how aggressively or not the model regresses to the mean, etc.); and we show how these biases can be intuitive or very counterintuitive, depending on properties of the model and data. We also illustrate in a case study on outlier handling how multiple biases can interact in complex ways; and we discuss implications of our results for learning the bitter lesson and building TSFMs. 1 Introduction Design decisions associated with modern machine learning models typically involve multiple trial-and-error iterations using one or more of many possible “knobs,” including model architectures, loss functions, and parameters and hyperparameters such as learning rate, batch size, patch size, and embedding strategy. A common workflow is to adjust these knobs based on intuitions about the model and then use available data, typically in the form of curated, well-established benchmarks, to train and evaluate the model. For example, in the case of time series, intuitions include that nearby data points are similar and should be mapped close to each other, that time is continuous and that this continuity should be maintained within the model, and that regression algorithms should regress to the mean. This process, which is expensive (since one typically has only weak control on the precise effect of varying knobs, since various knobs interact in complex ways, and since one often does not even know what knobs are most relevant for improving a given downstream task) is iterated upon until one obtains a model that is quantitatively “better” on well-established benchmarks. Unfortunately, model improvements often do not stand the test of time—when the data and compute are much larger, or when the data are qualitatively different than the well-established benchmarks on which the model was trained, the same knob fiddling can have a different, and often deleterious, effect. This leads to worse results and more model development effort in the longer term. Understanding how various design choices affect downstream model properties is of increasing importance as the community aims to design foundation-like models that are forward-compatible with qualitatively more and different data. Here, we consider these issues in the context of recently-developed time series foundation models (TSFMs). Our goal is not to develop “yet another” model. Rather, we aim to understand the effects, i.e., the implicit biases, encoded by these various design decisions. We are particularly interested in how those effects vary with the properties of the data and the model, in particular for data that are similar to well-established benchmarks versus data that are qualitatively different. In time series forecasting, there are multiple well-known public benchmarks, e.g., M4 ( Makridakis et al. 2020 ) , ETT ( Zhou et al. 2021 ) , and the large collection from Godahewa et al. 2021 including Kaggle’s Tourism ( Athanasopoulos et al. 2011 ) , Electricity ( Trindade 2015 ) , Traffic ( Lai et al. 2017 ) , and Dominick ( James M. Kilts Center 2020 ) . There is also a recent explosion of TSFMs, e.g., LLMTime ( Gruver et al. 2023 ) , Chronos ( Ansari et al. 2024a ) , Moirai ( Woo et al. 2024 ) , TimesFM ( Das et al. 2024 ) , Chronos-Bolt ( Ansari et al. 2024b ) , WaveToken ( Masserano et al. 2025 ) , and TOTEM ( Talukder et al. 2024 ) . If a foundation model is to be easily-adapted to various tasks, with minimal additional task-specific data ( Liang et al. 2024 ) , this leads to the important open question: how does one know the biases of various knobs in a TSFM on data that are qualitatively different than the benchmarks on which the TSFM was trained and evaluated? Figure 1 : An illustration of the three broad classes of biases that we identify and investigate. The temporal bias tends to be induced by patching, which transplants the temporal dependencies from the input sequence to a Transformer’s hidden space; the geometric bias tends to be related to the input embedding, where a discrete versus a continuous embedding preserves the input space’s geometry differently; the regression-to-the-mean bias tends to be caused by the training loss, where a continuous loss tends to average different outcomes more than a cross-entropy loss does. We also illustrate the interaction of these biases in a case study of outlier handling. In this paper, we identify three classes of design choices — the “knobs” of patch size, whether the embedding is discrete or continuous, and the choice of loss; and, based on extensive analysis, we characterize three corresponding classes of implicit biases — which we call the “temporal bias,” the “geometric bias,” and the “regression-to-the-mean bias.” We describe in detail how each design choice tends to affect the corresponding bias; and we show how, depending on the model, this effect could be intuitive or very counterintuitive. We also provide an example of how the design choices can interact in complex ways. Our main results, summarized in Figure 1 , are as follows. • Patch Size \rightarrow Temporal Bias. Given a discrete sequence sampled from a time series, many TSFMs group these inputs into patches of length k , with k=1 as a “trivial” corner case, and map each patch into a Transformer’s hidden space. This design choice of the patch size parameter leads to a temporal bias : depending on its value, a TSFM may favor low-frequency information more or less aggressively. A larger patch size, e.g., 16 in Chronos-Bolt, favors low-frequency components in the time series, which can be challenging on chaotic (or even non-chaotic) systems that consist of high frequency and/or mixed-frequency components. Conversely, a smaller patch size, e.g., 1 in Chronos or 2 or 4 or 8 in Chronos-Bolt, can capture parts of these high frequency components. Importantly, many time series are periodic or quasi-periodic, 1 1 1 Periodic phenomena repeat at precise intervals, while quasi-periodic phenomena follow complex, non-repeating patterns of roughly regular intervals. Examples of the former include New Year’s Day and someone’s birthday, which (up