TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning

Yaxuan Kong, Qingren Yao Yuqi Nie, Yichen Li, Yilei Shao, Stefan Zohren, Anna Vettoruzzo, Joaquin Vanschoren, Ming Jin, Qingsong Wen · 2026

Read TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning by Yaxuan Kong, Qingren Yao Yuqi Nie, Yichen Li, Yilei Shao, Stefan Zohren, Anna Vettoruzzo, Joaquin Vanschoren, Ming Jin, Qingsong Wen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01498.txt.

Introduction

Kong and co-authors benchmark agents on multi-turn time series reasoning. Time series inform key decisions in many real domains. Agents can analyze data through language and tools. But reliable analysis across a long conversation is unproven. Past benchmarks test single-step tasks like forecasting. Real workflows evolve as user goals shift over turns. Conclusions emerge from accumulated evidence in those workflows. TimeSage-MT holds 240 tasks and 2,680 dialogue turns.

Eight real-world domains span basic exploration to decisions. A reproducible pipeline builds the tasks from real data. Answers stay verifiable by construction in the pipeline. Frontier models show sharp drops on decision-oriented tasks. Memory failures drive much of that drop. Uncertainty handling and domain decisions also fail often.

Catalog record

CREATORYaxuan Kong, Qingren Yao Yuqi Nie, Yichen Li, Yilei Shao, Stefan Zohren, Anna Vettoruzzo, Joaquin Vanschoren, Ming Jin, Qingsong Wen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01498

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning Yaxuan Kong, Qingren Yao Yuqi Nie, Yichen Li, Yilei Shao, Stefan Zohren, Anna Vettoruzzo, Joaquin Vanschoren, Ming Jin, Qingsong Wen Abstract Time series data inform critical decisions across many real-world domains. While large language model (LLM) agents can analyze data through natural language and tools, it remains unclear whether they can conduct reliable time series analysis across multi-turn conversations. Existing benchmarks focus on single-step tasks such as forecasting and anomaly detection, overlooking practical workflows where user goals evolve, agents must build on prior analyses, and conclusions emerge from accumulated evidence. In this work, we introduce TimeSage-MT, a multi-turn benchmark for agentic time series reasoning with 240 tasks and 2,680 dialogue turns across 8 real-world domains, spanning basic exploration to decision-oriented analysis. TimeSage-MT is built through a reproducible pipeline that converts real-world time series data into multi-turn conversations with verifiable answers. It provides a unified evaluation protocol and public leaderboard for comparing time series agentic systems. To demonstrate the benchmark’s utility, we evaluate frontier LLMs alongside TimeSage, a novel structured agent equipped with a comprehensive time series skill library. The results show sharp performance drops on decision-oriented tasks, driven by failures in memory, uncertainty handling, and domain-based decision making. TimeSage-MT exposes critical gaps in current agentic reasoning and provides a rigorous foundation for future development. 1 Introduction Time series analysis drives critical decision-making across high-stakes domains, from financial portfolio balancing to healthcare risk assessment 5 ; 14 ; 17 . Time series data capture how systems evolve, when risks arise, and how actions affect future outcomes 25 ; 21 ; 35 . In practice, however, useful time series analysis rarely ends with a single prediction 18 ; 9 . Analysts often inspect data quality, identify temporal structure, choose appropriate methods, quantify uncertainty, and translate results into actions that respect domain constraints 42 ; 32 . A financial analyst may assess volatility and regime shifts before deciding whether to rebalance a portfolio, and a grid operator may combine load forecasting, anomaly detection, and uncertainty analysis before making an operational recommendation. This workflow exposes a gap between existing time series benchmarks and the reasoning behavior expected from general-purpose analytical agents. Standard benchmarks for forecasting, anomaly detection, and classification have made important progress, but typically evaluate isolated tasks with fixed objectives and task-specific metrics 11 ; 23 ; 28 . They do not test whether an agent can maintain the dialog state, select analytical methods, chain dependent computations, adapt when assumptions fail, or communicate conclusions in a domain-based form 38 ; 36 ; 6 . As LLM agents become increasingly capable of natural language interaction, code execution, tool use, planning, and self-correction, this evaluation gap becomes urgent 22 ; 43 . Without benchmarks that evaluate the entire analytical trajectory, it is hard to distinguish reliable time series reasoning from shallow pattern matching, hallucinated numbers, accidental tool use, or plausible but unsupported explanations 39 . Designing such a benchmark is challenging for three reasons. First , time series reasoning is inherently multi-capacity 42 . As shown in Figure 1 , an effective agent must read the data properties ( C1 ), select suitable methods ( C2 ), configure the analyses correctly ( C3 ), remember prior results across turns ( C4 ), chain dependent analyzes ( C5 ), adapt when assumptions fail or user goals change ( C6 ), make actionable decisions ( C7 ), calibrate uncertainty ( C8 ) and ground conclusions in domain knowledge ( C9 ). Second , the depth of reasoning varies substantially between interactions. Some conversations require only exploratory profiling and summarization, while others require long decision-oriented workflows that combine forecasting, anomaly detection, risk assessment, synthesis, and justification. Third , evaluation must be reproducible and interpretable. Multi-turn agent outputs can appear convincing even when the underlying analysis violates assumptions, ignores earlier turns, or relies on unsupported reasoning 39 ; 32 . A meaningful benchmark therefore needs diverse real-world datasets, realistic multi-turn tasks, verifiable answer targets, and a unified protocol that supports both end-to-end evaluation and fine-grained diagnosis across capabilities, domains, and task types. To address these challenges, we introduce TimeSage-MT , a multi-turn benchmark and evaluation suite for agentic time series reasoning. TimeSage-MT contains 240 tasks and 2,680 dialog turns across 8 real-world domains , organized into 4 complexity levels : open exploration, multi-skill analysis, grounded synthesis, and full decision paths. Each task is paired with analysis trajectories, reference code, numerical ground truth, and level-specific output requirements, enabling evaluation of both final answers and intermediate reasoning behavior. To ensure scalability without sacrificing rigor, we engineer a reproducible pipeline that transforms raw datasets into verifiable, multi-turn analytical trajectories. TimeSage-MT further provides a unified evaluation protocol and leaderboard to compare time series agentic systems. To demonstrate the benchmark’s utility, we evaluate six frontier LLMs and multiple reasoning paradigms (direct, code-enabled, and skill-guided), alongside TimeSage, a novel structured agent system equipped with a comprehensive time series skill library comprising 226 analytical skills across 36 categories . By comparing these settings, TimeSage-MT exposes critical gaps in current agentic reasoning from basic exploration to decision-oriented analysis. Experiments show that current systems can often handle short exploratory tasks, but performance degrades sharply when tasks require multi-turn memory, uncertainty-aware reasoning, and domain-grounded decisions. These observations reveal failures not captured by single-task benchmarks and establish TimeSage-MT as a rigorous foundation for future development. Figure 1 : Overview of TimeSage-MT. It organizes 240 tasks over 4 levels and targets 9 capabilities for agentic time series reasoning. Filled circles mark active capabilities; dashes mark inactive ones. We release the dataset at https://huggingface.co/datasets/TimeSage-Series/TimeSage-MT and the code at https://github.com/TimeSage-Series/TimeSage-MT . Our main contributions are summarized below: • The first multi-turn benchmark for agentic time series reasoning. To the best of our knowledge, TimeSage-MT is the first benchmark for evaluating agentic time series reasoning across multi-turn analytical workflows. It contains 240 tasks and 2,680 dialogue turns across 8 domains and 4 complexity levels, from open exploration to full decision paths. Each task includes analysis trajectories, reference code, numerical ground truth, and level-specific outputs, and is organized around a 9-capability taxonomy to diagnose failures beyond the accuracy of the final-answer. • A reproducible end-to-end construction and pipeline. We develop an end-to-end pipeline that converts real-world time series datasets into annotated multi-turn conversations with verifiable answer targets. The pipeline covers dataset collection, conversation generation, trajectory construction, reference code generation, numerical verification, and human annotation. We also release a reusable annotation dashboard to support task review and future benchmark extension. • A unified evaluation protocol, reusable skill library, and key observations. We provide a public leaderboard and evaluation protocol that assess agentic systems by per-turn answer accuracy and capability-level reasoning behavior. We release a reusable library of 226 analytical skills across 36 categories, an evaluation dashboard with per-turn traces and scores, and an open platform for multi-turn interaction with time series data. Under this protocol, we compare frontier LLM-based settings, including direct response, code execution, skill-guided analysis, and TimeSage, a structured agentic system built on our skill library. Our results reveal key observations that point to persistent failures in memory, uncertainty-aware reasoning, and domain-based decision-making. 2 Related Work Table 1 : Comparison with representative prior benchmarks.

Read the complete text