From Tokenizer Bias to Backbone Capability: A Controlled Study of LLMs for Time Series Forecasting
Xinyu Zhang, Shanshan Feng, Xutao Li, Kenghong Lin, Fan Li, Pengfei Jia · 2025
Read From Tokenizer Bias to Backbone Capability: A Controlled Study of LLMs for Time Series Forecasting by Xinyu Zhang, Shanshan Feng, Xutao Li, Kenghong Lin, Fan Li, Pengfei Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08818.txt.
Introduction
Xinyu Zhang and co-authors run a controlled test of whether language models truly help forecast time series, in a 2025 paper. A popular recipe chops a series into patches, translates them into model tokens, and reads forecasts back out. On small data sets, the translator parts can memorize the data and hide the model's real skill. The team built three models that differ only in how they were pre-trained.
Large-scale pre-training gave translator pairs that do not lean on one data set's habits. With those in place, the language model backbone showed some forecasting promise in zero-shot and few-shot tests. Its results still trailed models trained directly on large time series collections. The authors share their code and urge care in judging what the backbone itself contributes.
Catalog record
| CREATOR | Xinyu Zhang, Shanshan Feng, Xutao Li, Kenghong Lin, Fan Li, Pengfei Jia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08818
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Tokenizer Bias to Backbone Capability: A Controlled Study of LLMs for Time Series Forecasting Xinyu Zhang, Shanshan Feng, Xutao Li, Kenghong Lin, Fan Li, Pengfei Jia Abstract . Using pre-trained large language models (LLMs) as a backbone for time series prediction has recently attracted growing research interest. Existing approaches typically split time series into patches, map them to the token space of LLMs via a Tokenizer , process the tokens through a frozen or fine-tuned LLM backbone, and then reconstruct numerical forecasts using a Detokenizer . However, the actual effectiveness of LLMs for time series forecasting remains under debate. We observe that when trained and evaluated on small datasets, these Tokenizer–Detokenizer pairs often overfit to the specific data distribution, thereby masking the intrinsic predictive capability of the LLM backbone. To investigate the inherent potential of LLMs in this context, we design three models with identical architectures but distinct pre-training strategies. By leveraging large-scale pre-training, we obtain more unbiased Tokenizer–Detokenizer pairs that are seamlessly integrated with the LLM backbone. Through controlled experiments, we evaluate the zero-shot and few-shot forecasting performance of the LLM, offering insights into its true capabilities. Our extensive experiments reveal that, although the LLM backbone shows some promise, its performance remains limited and does not consistently surpass that of models specifically trained on large-scale time series data. Our source code is publicly available in the repository: https://github.com/SiriZhang45/LLM4TS. Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1; August 9–13, 2026; Jeju Island, Republic of Korea. Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (KDD 2026), August 9–13, 2026, Jeju Island, Republic of Korea ISBN: 979-8-4007-2258-5/2026/08 DOI: 10.1145/3770854.3780318 CCS: Applied computing Forecasting CCS: Computing methodologies Neural networks † † cc-license: by 1. Introduction Time series forecasting is a long-standing and essential task with applications across numerous fields, including weather forecasting, renewable energy power prediction, stock market forecasting, and disease spread prediction. The evolution of time series forecasting models has progressed from early statistical methods like AutoRegressive Integrated Moving Average (ARIMA) ( Box et al., 2015 ) to advanced deep learning architectures such as RNNs ( Salinas et al., 2020 ) , LSTMs, and CNNs ( Wang et al., 2022 ; Wu et al., 2022 ) , culminating in the latest, powerful Transformer-based models ( Nie et al., 2022 ) and other networks ( Zhang et al., 2024 ) . This progression has significantly improved the accuracy and performance of predictive models in various domains. In natural language processing, the development of large language models (LLMs) such as GPT ( Radford et al., 2019 ) and LLaMA ( Touvron et al., 2023 ) has led to recent advancements demonstrating impressive few-shot and zero-shot capabilities in both linguistic and vision-language tasks. Consequently, performing time series prediction tasks based on pre-trained LLMs has emerged as a novel research trend within the time series prediction community ( Zhou et al., 2023 ; Tan et al., 2024 ; Hu et al., 2025 ) . These works focus on adapting pre-trained LLMs to classify, forecast, and detect anomalies in time series data. These studies assume that language models, sophisticated models of sequential dependencies in text, can generalize to sequential dependencies in time series data. Early efforts in leveraging LLMs for time series prediction primarily involved converting numerical values into their character forms and directly feeding them into LLMs to predict future numeric sequences ( Xue and Salim, 2023 ; Gruver et al., 2024 ) . More recent advances, however, have shifted toward a more structured paradigm that aligns with conventional time series modeling workflows. In this prevailing approach, time series are divided into patches, which are then tokenized into the token dimensions of LLMs. These tokens are processed through a frozen or fine-tuned LLM backbone, and the final output is detokenized into numerical predictions for future time points ( Zhou et al., 2023 ) . Empirical results on public datasets demonstrate that this paradigm can achieve performance comparable to or even surpassing that of state-of-the-art (SOTA) time series forecasting models. In this paper, we focus on this mainstream paradigm. Table 1 . Comparison of Existing Methods and Our Approach in Time Series Forecasting Using LLMs Studies Affirming LLM Effectiveness ( Zhou et al., 2023 ; Jin et al., 2023 ; Jia et al., 2024 ; Liu et al., 2024c ) Studies Refuting LLM Effectiveness ( Tan et al., 2024 ) Ours Training Data Scale Small datasets Small datasets Large-scale pre-training dataset Testing Setup In-distribution, on training-aligned data In-distribution, on training-aligned data Zero-shot prediction (out-of-distribution) Tokenizer-Detokenizer Coupling Strong coupling with backbone Strong coupling with backbone Decoupled Evaluation of LLM Effectiveness ✗ No quantification ✗ No quantification ✓ Quantified Despite the proliferation of methods applying LLMs to time series forecasting, the actual efficacy of LLMs in this domain remains controversial. A recent study ( Tan et al., 2024 ) conducted extensive ablation experiments in which the pre-trained LLM backbone was replaced with simpler alternatives—such as standard Transformer or attention blocks—or even removed entirely. Surprisingly, performance on small datasets remained comparable, and in some cases even improved. The authors attributed these results to architectural factors such as the patching mechanism and channel independence, suggesting that the pre-trained LLM backbone may not be essential for achieving strong forecasting performance. However, these findings do not conclusively demonstrate that LLMs are ineffective. Notably, models incorporating frozen LLMs still achieved or approached SOTA performance on several benchmarks, indicating that pre-trained knowledge may still play a beneficial role. This leaves open the question of whether—and to what extent—LLMs contribute to the observed performance gains. Indeed, several follow-up studies continue to operate under the assumption that LLMs possess inherent capabilities for time series prediction ( Wolff et al., 2025 ; Liu et al., 2024b ; Liu et al., 2025 ; Hu et al., 2025 ) . Therefore, while the work of Tan et al. ( Tan et al., 2024 ) raises important concerns about the necessity of LLMs in time series forecasting, a deeper examination of their experimental design and conclusions is warranted. A more nuanced understanding is needed to disentangle the contributions of model architecture from those of pre-trained representations. We argue that the common practice in existing studies—training and testing on small datasets using a frozen LLM backbone, with the Tokenizer and Detokenizer trained solely on limited data—is inadequate for assessing the true potential of frozen Transformers in time series forecasting. Since the Tokenizer and Detokenizer can adapt effectively to the small dataset without relying on pre-trained knowledge, they become highly specialized to the small training data. This tight coupling with the specific dataset masks the intrinsic predictive capability of the LLM backbone, thereby undermining the validity of such evaluations. Due to the limitations of conventional train-then-test evaluation on small datasets, zero-shot and few-shot assessment methods offer a more appropriate framework for evaluating LLM capabilities in time series forecasting. However, existing works on such evaluations still suffer from critical flaws ( Zhou et al., 2023 ; Tan et al., 2024 ) . Typically, these methods train the Tokenizer and Detokenizer on a single small dataset with a frozen LLM backbone, then test on a different dataset. When trained on limited