When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inference

Wen Ye, Jinbo Liu, Defu Cao, Wei Yang, Yan Liu · 2025

Read When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inference by Wen Ye, Jinbo Liu, Defu Cao, Wei Yang, Yan Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01822.txt.

Introduction

Wen Ye, Jinbo Liu, Defu Cao, Wei Yang, and Yan Liu test whether large language models can reason across time series data in many steps. Time series are values tracked over time, like sales or sensor reads. Real tasks need chained steps, such as forecasting under limits or spotting faults with tuned thresholds. The team builds a benchmark drawn from more than twenty published task designs.

Their bench holds thirty-three real world task types with a generator that can grow over time. Each task type carries its own success rules and quality measures. Eight leading models are run under one shared protocol. All struggle to assemble full workflows for time data tasks. The bench is public, giving builders a shared yardstick for progress.

Catalog record

CREATORWen Ye, Jinbo Liu, Defu Cao, Wei Yang, Yan Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01822

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inference Wen Ye, Jinbo Liu, Defu Cao, Wei Yang, Yan Liu Abstract The rapid advancement of Large Language Models (LLMs) has sparked growing interest in their application to time series analysis tasks. However, their ability to perform complex reasoning over temporal data application domain remains significantly underexplored. To achieve the goal, one first step is to establish a rigorous benchmark dataset for evaluation. In this work, we introduce TSAIA Benchmark, a first attempt to evaluate LLMs as time series artificial intelligence assistant. To ensure both scientific rigor and practical relevance, we surveyed over 20 academic publications and identified 33 real-world task formulations. The benchmark encompasses a broad spectrum of challenges, ranging from constraint-aware forecasting to anomaly detection with threshold calibration—tasks that require compositional reasoning and multi-step time series analysis. The question generator is designed to be dynamic and extensible, supporting continuous expansion as new datasets or task types are introduced. Given the heterogeneous nature of the tasks, we adopt task-specific success criteria and tailored inference quality metrics to ensure meaningful evaluation for each task. We apply this benchmark to assess 8 state-of-the-art LLMs under a unified evaluation protocol. Our analysis reveals limitations in current models’ ability to assemble complex time series analysis workflows, underscoring the need for specialized methodologies for adaptation towards domain-specific applications. Our benchmark is available at https://huggingface.co/datasets/Melady/TSAIA , and the code is available at https://github.com/USC-Melady/TSAIA . 1 Introduction Large Language Models (LLMs) have demonstrated remarkable general-purpose capabilities across a range of domains, including language understanding dong2019unified , code generation jiang2024survey , and scientific reasoning taylor2022galactica . Despite their growing capabilities, large language models have not been evaluated for time series analysis, which constitutes a fundamental competency for data analysts and scientists across critical domains including energy alvarez2010energy , finance sezer2020financial , climate science mudelsee2019trend , and healthcare rathlev2007time . In practice, time series workflows in the real world are inherently complex yan2021learning ; han2021real : they demand multi-step reasoning fu2022complexity , precise numerical computation cvejoski2022future , integration of domain knowledge xue2024domain , and adherence to operational constraints wang220kv . With the advent of agent systems, there is growing interest among practitioners in developing intelligent agents capable of interpreting natural language instructions to perform time series analysis. However, given that time series understanding poses challenges even for humans, researchers are striving to establish a clear paradigm for how LLMs can effectively contribute to these complex downstream tasks. A crucial step toward empowering LLMs in time series applications is the development of rigorous benchmark datasets for comprehensive evaluation. Despite recent interest in evaluating LLMs’ ability to solve temporal-related tasks merrill2024language , existing benchmarks are insufficient for evaluating whether LLMs can function as general-purpose time series inference agents capable of constructing complete, constraint-aware analytical workflows. Most focus on individual tasks du2024tsi and often evaluated under fixed experiment configuration (using 336 historical timesteps to predict the next 96 timesteps zhou2021informer ), direct QA questions that only involve temporal concepts but do not involve time series or require numerical precision ability wang2024tram , and lack of coverage on real-world operational constraints. Critically, they fail to capture whether LLMs can serve as a general-purpose time series assistant. To push the boundary of LLM-based time series agents and address the lack of a suitable benchmark, we introduce T ime S eries A rtificial I ntelligence A ssistant TSAIA Benchmark. The benchmark is grounded in practical relevance ( what tasks are suitable? ), enables dynamic extensibility ( how to generate specific task instances? ), and facilitates unified evaluation ( how to perform evaluation over heterogeneous task types? ). We collected and curated diverse datasets from energy, climate science, finance, and healthcare domains. Drawing from a review of over 20 time series application publications, identified 33 common tasks types adding up to 1054 questions in total, covering predictive, diagnostic, analytical, and decision-making tasks. To achieve good performance on TSAIA, the following capabilities are needed: compositional reasoning li2024understanding (the sequential execution of logical and numerical operations to construct end-to-end analytical pipelines), comparative reasoning (selecting the optimal asset based on calculated summary indicators), commonsense reasoning davis2015commonsense (identifying plausible covariates for the target variable), decision-oriented reasoning (interpreting risk-return metrics in investment contexts), and numerical precision. We evaluate eight state-of-the-art models: GPT-4o hurst2024gpt , Qwen2.5-Max qwen2025max , Llama-3.1 Instruct 70B grattafiori2024llama , Claude-3.5 Sonnet anthropic2024claude35sonnet , DeepSeek liu2024deepseek , Gemini-2.0 google2024gemini2 , Codestral mistral2024codestral , and DeepSeek-R guo2025deepseek —using the CodeAct agent framework wang2024executable implemented via AgentScope agentscope . Each agent is prompted to generate executable Python code to perform task-specific inference, with iterative refinement based on execution feedback. This approach accommodates LLM limitations on premature output termination for long sequence output song2025hansel and tokenizing numerical values to discrete tokens spathis2024first . Our findings reveal that while certain models show strengths on narrow task types, none reliably generalizes across the full benchmark. Common failure modes include inadequate numerical result/trivial prediction and the inability to assemble complex workflows. These results underscore the challenges of structured numerical reasoning in real-world time series settings and position TSAIA as a critical benchmark for future development of domain-grounded, reasoning-capable time series AI assistants. Figure 1: Categorization of Tasks in TSAIA. Lighter colors denote tasks with less difficulty and darker colors denote tasks with higher difficulty. 2 TSAIA Benchmark 2.1 Task Annotation& Data Collection: What time series tasks hold real world practicality? To evaluate time series AI assistants effectively, we focus on tasks grounded in real-world use cases that data analyst in different time series application domains may face. We aim for the tasks to assess an assistant’s ability to handle practical scenarios involving time series data, ensuring their utility in everyday analytical workflows. We start by forming a group of three graduate students with backgrounds in machine learning and applied time series analysis to collect real world applications of time series analysis that exhibit multi-step complexity and requires reasoning. We reviewed over 20 research publications addressing time series analysis challenges, extracting and converting core problems into benchmark tasks. The collected tasks can be categorized into four broad groups based on the nature of the underlying problem as shown in figure 1 : (1) Predictive Tasks involve forecasting with or without covariates, with constraint-aware prediction that must comply with real-world operational requirements such as maximum/minimum load limits wang220kv ; greif1999short , ramp rate constraints schaible2024application ; gkavanoudis2023provision , and variability thresholds han2021real . (2) Diagnostic Tasks focus on identifying abnormal patterns or latent structures within the data. This includes anomaly detection tasks, which may leverage reference samples or known anomaly frequencies yan2021learning ; song2024unsupervised ; niu2020lstm ; verma2023deep ; and causal discovery tasks, where models infer binary causal graphs from observational time series with domain-specific priors such as partial causal ratios or qualitative relationships aoki2020parkca . (3) Analytical Tasks target analytics based on time series trends, particularly within financial domains. These tasks may require performing risk-return analysis riley2022maximum ; miglietti2020bitcoin or generating trading strategies using historical asset prices shalini2019picking ; pramudya2020technical . (4) Decision-Making Tasks are framed as multiple-choice questions, primarily in the financial domain. These require models to analyze structured summaries such as portfolio performance under different metrics steinki2015common or compare a

Read the complete text