TimeSeriesGym : A Scalable Benchmark for (Time Series) Machine Learning Engineering Agents
Yifu Cai *, Xinyu Li *, Mononito Goswami *, Michał Wiliński, Artur Dubrawski · 2025
Read TimeSeriesGym : A Scalable Benchmark for (Time Series) Machine Learning Engineering Agents by Yifu Cai *, Xinyu Li *, Mononito Goswami *, Michał Wiliński, Artur Dubrawski (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13291.txt.
Introduction
Yifu Cai, Xinyu Li, Mononito Goswami, Michał Wiliński, Gus Welter, and Artur Dubrawski built TimeSeriesGym, a benchmark for AI agents that do machine learning engineering on time series. Older benchmarks test narrow model building and grade only a final submission file. Real engineering spans data handling, reading research code, and translating code between tools. TimeSeriesGym draws challenges from many domains to test single skills and their combinations.
The framework grades several artifacts at once: submission files, code, and trained models. Scoring mixes exact numeric measures with LLM-based judgment for flexibility. Tools in the framework let designers spin up many challenges at scale, and the design can extend past time series.
Catalog record
| CREATOR | Yifu Cai *, Xinyu Li *, Mononito Goswami *, Michał Wiliński, Artur Dubrawski |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13291
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TimeSeriesGym : A Scalable Benchmark for (Time Series) Machine Learning Engineering Agents Yifu Cai *, Xinyu Li *, Mononito Goswami *, Michał Wiliński, Artur Dubrawski Abstract We introduce TimeSeriesGym , a scalable benchmarking framework for evaluating Artificial Intelligence (AI) agents on time series machine learning engineering challenges. Existing benchmarks lack scalability, focus narrowly on model building in well-defined settings, and evaluate only a limited set of research artifacts (e.g., CSV submission files). To make AI agent benchmarking more relevant to the practice of machine learning engineering, our framework scales along two critical dimensions. First, recognizing that effective ML engineering requires a range of diverse skills, TimeSeriesGym incorporates challenges from diverse sources spanning multiple domains and tasks. We design challenges to evaluate both isolated capabilities (including data handling, understanding research repositories, and code translation) and their combinations, and rather than addressing each challenge independently, we develop tools that support designing multiple challenges at scale. Second, we implement evaluation mechanisms for multiple research artifacts, including submission files, code, and models, using both precise numeric measures and more flexible LLM-based evaluation approaches. This dual strategy balances objective assessment with contextual judgment. Although our initial focus is on time series applications, our framework can be readily extended to other data modalities, broadly enhancing the comprehensiveness and practical utility of agentic AI evaluation. We open-source our benchmarking framework to facilitate future research on the ML engineering capabilities of AI agents. 1 Introduction AI agents [ 9 , 11 ] have shown growing promise in automating machine learning (ML) and data science (DS) workflows. Fueled by powerful large language models (LLMs), they can reason about context, adapt to diverse tasks, and iteratively refine their solutions over long horizons. Such capabilities have the potential to significantly reduce mundane, mostly manual efforts required in ML engineering and improve the overall productivity of ML practice. To measure progress in this area, several benchmarks [ 9 , 3 , 12 , 17 , 22 , 1 , 20 , 18 ] have been introduced to evaluate the capabilities of AI agents on ML and DS tasks. However, existing benchmarks have important limitations. First, many of them source ML challenges primarily from popular competitions such as Kaggle, which are well-structured and do not fully capture the complexity of real-world ML tasks. Second, evaluations are typically outcome-based, focusing on overall task completion or eventual model performance metrics such as accuracy, while combining and obfuscating the impact of multiple component skills that jointly determine the outcomes, such as effective data wrangling or code quality improvement capabilities. Third, current benchmarks lack scalability, as tasks have to be manually curated and cannot be developed at scale. To enable efficient evaluation of AI agents in realistic ML scenarios, we propose TimeSeriesGym , a scalable and agent-agnostic benchmarking framework for evaluating AI agents on time series ML engineering tasks. TimeSeriesGym currently consists of 34 challenges that span 8 unique time series problems (forecasting, classification, time series understanding), from more than 15 domains (healthcare, finance, epidemiology). Our benchmark covers both Kaggle-style challenges and original tasks carefully designed based on real-world ML engineering practice. While we focus on time series analytics due to its prevalence in applications and under-representation in existing agent benchmarks, our framework is modality-agnostic in principle and can be easily extended to handling other data modalities (e.g., images, text, audio) via the accompanying tools for scalable task generation. TimeSeriesGym provides an interactive gym environment compatible with various types of agent scaffolding, allowing seamless evaluation of agents of different types and collection of their trajectories. Beyond benchmarking, this allows TimeSeriesGym to also serve as a data flywheel for future agent improvement through post-training using the collected trajectory data. Our contributions include: 1. We propose TimeSeriesGym , the first open source benchmark environment for evaluating AI agents on time series ML engineering tasks (Fig. 1 ). 2. TimeSeriesGym offers a scalable task generation mechanism that reduces manual efforts in task curation and ensures long-term sustainability of the benchmark. 3. TimeSeriesGym provides a comprehensive framework that evaluates multimodal agent outputs (e.g., prediction files, models, code) across specific ML skills (e.g., data handling, model improvement), using a holistic approach that combines quantitative metrics (e.g., accuracy) with qualitative assessment (e.g. LLM-as-a-judge for code utility). Figure 1: TimeSeriesGym is a scalable benchmarking environment for ML engineering agents. It currently features 34 time series challenges across 8 unique time series problems, spanning more than 15 domains. Challenges are either carefully designed based on real-world ML practice, or sourced from Kaggle competitions and GitHub repositories. TimeSeriesGym includes key mechanisms to enable efficient and scalable generation of new challenges. Our evaluation methodology combines precise quantitative metrics with flexible qualitative assessment, and provides specialized tools to grade various artifacts generated during ML engineering. TimeSeriesGym is compatible with many different agent types, even those with fundamentally distinct designs. 2 Related Work Machine learning agent benchmarks. Several benchmarks have been proposed to evaluate LLM agents on automating ML and DS tasks. These benchmarks are typically structured around three key components: (1) task curation, (2) agent capabilities being evaluated, and (3) evaluation protocol. Benchmarks differ in how they curate ML/DS tasks. For example, MLE-bench [ 3 ] and DSBench [ 12 ] compile tasks from online competition platforms such as Kaggle, while other benchmarks source tasks from ML-related Github repositories [ 1 , 20 ] or hand-craft tasks based on ML research problems or engineering workflows [ 9 , 17 , 22 , 18 ] . With regard to agent capabilities, some benchmarks [ 3 , 9 , 17 , 22 , 12 ] focus on comprehensive ML science skills by evaluating agents on end-to-end problem solving skills, while others [ 1 , 20 , 18 ] focus on more modular engineering-oriented capabilities within the ML pipeline, such as using GitHub repositories or integrating APIs. Evaluation protocols also differ in output formats and granularity. MLE-bench [ 3 ] and DSBench [ 12 ] require agents to output results in specific formats (e.g., CSV files) that can be directly scored using predefined metrics such as accuracy, while other benchmarks [ 9 , 17 , 22 ] allow for more flexible outputs in addition to prediction files, such as model artifacts and code. ML-Dev-Bench [ 18 ] further extends the evaluation by specific skills (e.g., data handling, model improvement), while SUPER [ 1 ] provides a more holistic evaluation by combining outcome-based evaluation with qualitative code inspection to assess agents’ progress towards completing the tasks. Table 1: Comparison of TimeSeriesGym with existing ML/DS agent benchmarks. Categories include Number: total and time series (TS) tasks in each benchmark, where each task corresponds to a unique data source (e.g., a single Kaggle competition or GitHub repository); Source: task origins (K: Kaggle, G: GitHub, H: Hand-crafted); ML Capability: coverage of ML science tasks (e.g., modeling, open-ended research) and engineering tasks (e.g., repository utilization, API integration); and Evaluation: capabilities for evaluating multimodal outputs (e.g., prediction files, model artifacts), specific ML skills (e.g., data handling, model improvement), and from a holistic perspective (combining quantitative metrics (accuracy, mean absolute error) with qualitative evaluation via code landmarks or LLM-as-a-judge approaches). We use "+" to indicate TimeSeriesGym ’s scalability which enables the generation of an unlimited number of new