ML-Tool-Bench: Tool-Augmented Planning for ML Tasks
Yaswanth Chittepu, Raghavendra Addanki, Tung Mai, Anup Rao, Branislav Kveton · 2025
Read ML-Tool-Bench: Tool-Augmented Planning for ML Tasks by Yaswanth Chittepu, Raghavendra Addanki, Tung Mai, Anup Rao, Branislav Kveton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00672.txt.
Introduction
Yaswanth Chittepu, Raghavendra Addanki, Tung Mai, Anup Rao, and Branislav Kveton present ML-Tool-Bench in a 2025 paper. The bench is a test ground for machine learning agents that must do a full data science job on their own. Such agents chain many steps together, from first look at data to final model.
Along the way, an agent must clean data, build features, pick a model, and tune its settings. Each choice affects the next, so planning and revision matter. A shared bench lets researchers compare agents on the same tasks and tools. That makes progress in autonomous data science easier to measure and harder to fake.
Catalog record
| CREATOR | Yaswanth Chittepu, Raghavendra Addanki, Tung Mai, Anup Rao, Branislav Kveton |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 118 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-00672
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ML-Tool-Bench: Tool-Augmented Planning for ML Tasks Yaswanth Chittepu, Raghavendra Addanki, Tung Mai, Anup Rao, Branislav Kveton Abstract The development of autonomous machine learning (ML) agents capable of end-to-end data science workflows represents a significant frontier in artificial intelligence. These agents must orchestrate complex sequences of data analysis, feature engineering, model selection, and hyperparameter optimization, tasks that require sophisticated planning and iteration. While recent work on building ML agents has explored using large language models (LLMs) for direct code generation, tool-augmented approaches offer greater modularity and reliability. However, existing tool-use benchmarks focus primarily on task-specific tool selection or argument extraction for tool invocation, failing to evaluate the sophisticated planning capabilities required for ML Agents. In this work, we introduce a comprehensive benchmark for evaluating tool-augmented ML agents using a curated set of 61 specialized tools and 15 tabular ML challenges from Kaggle. Our benchmark goes beyond traditional tool-use evaluation by incorporating an in-memory named object management, allowing agents to flexibly name, save, and retrieve intermediate results throughout the workflows. We demonstrate that standard ReAct-style approaches struggle to generate valid tool sequences for complex ML pipelines, and that tree search methods with LLM-based evaluation underperform due to inconsistent state scoring. To address these limitations, we propose two simple approaches: 1) using shaped deterministic rewards with structured textual feedback, and 2) decomposing the original problem into a sequence of sub-tasks, which significantly improves trajectory validity and task performance. Using GPT-4o, our approach improves over ReAct by 16.52 percentile positions, taking the median across all Kaggle challenges. We believe our work provides a foundation for developing more capable tool-augmented planning ML agents. 1 Introduction Autonomous agents capable of solving end-to-end machine learning (ML) tasks represent a critical frontier in artificial intelligence [ 9 , 28 , 32 , 3 ] . Such agents must be capable of doing: data preprocessing, feature engineering, model training, and hyperparameter tuning, while managing intermediate results and adapting their strategies based on the evolving context. Achieving this level of autonomy requires not only sophisticated planning, but also memory management and the capacity to coordinate multiple operations coherently. Large language models (LLMs) have recently been explored as the foundation for such agents [ 9 , 3 , 12 ] . Early work has primarily focused on direct code generation, where the agent generates python code for completing a given ML task [ 9 , 3 , 12 , 28 ] . This paradigm has shown promise on competitive benchmarks inspired by Kaggle challenges, with some approaches achieving performance comparable to a Kaggle Master [ 9 , 3 ] . Several benchmarks have also been proposed to evaluate the performance of LLMs on such tasks [ 3 , 12 , 21 , 14 , 38 ] . However, any approach that relies on direct code generation is prone to key weaknesses: generated code is brittle [ 1 , 17 ] , debugging typically requires multiple iterations, and reasoning is tightly coupled with execution [ 17 , 4 ] . An alternative paradigm equips LLMs with external tools, yielding tool-augmented agents that need to decide which tools to invoke and in what sequence, to solve the task. Tools offer modular, reusable building blocks for data-science workflows: from preprocessing, to training, and evaluation. This design has proven effective in broader domains, including web navigation [ 42 ] , operating systems [ 2 ] , and code interpretation [ 12 ] , yet its potential for ML workflows remains underexplored. Crucially, tool augmentation reformulates the problem as planning in a large action space: the agent must coordinate multi-step trajectories and retrieve and reuse intermediate artifacts (or results). Because the agent is restricted to a curated toolset, tool-augmented approaches decouple high-level reasoning from low-level code execution, improving modularity, reliability, and safety. Existing benchmarks for tool use fall short on long - horizon planning. Most benchmarks and approaches evaluate whether agents can select the right tools and valid arguments. The Berkeley Function - Calling Leaderboard (BFCL) [ 18 ] measures single, parallel, and multiple function calling, and BFCL - v3 [ 18 ] extends this to multi - turn, multi - step settings. However, even BFCL - v3 emphasizes relatively shallow plans compared to ML workflows, which might require long - term planning, iterative refinement, and reuse of intermediate artifacts. Similarly ToolBench [ 31 ] provides a suite of diverse software tools, that span both single-step and multi-step action generation, but focuses on evaluating whether the LLM can correctly select tools and tool arguments. In this work, we introduce ML - Tool - Bench, motivated by the lack of good benchmarks to assess planning approaches with tools in ML workflows. In particular, ML - Tool - Bench provides a benchmark to evaluate the planning capabilities of LLM agents on tabular Kaggle ML challenges. We introduce a curated suite of 61 tools sufficient to solve such tasks and assess performance across 15 Kaggle challenges spanning regression and classification. We evaluate multiple agents using several different planning algorithms, on our benchmark. To enable agents to create, persist, and reuse intermediate artifacts, we adopt an in - memory, named - object management scheme: tools accept references to named objects, and agents can assign names to tool outputs. We refer to this as scratchpad - augmented planning: agents store and retrieve objects by name over multi - step trajectories, enabling tools to handle arbitrarily large or structured inputs, unlike prior benchmarks that restrict arguments to simple types (e.g., strings, integers, floats). We observe that simple methods like ReAct [ 35 ] struggle to produce performant trajectories across our Kaggle benchmark. Monte Carlo Tree Search-based methods [ 15 , 24 ] such as LATS [ 41 ] , which rely on LLMs as value estimators, also underperform due to inconsistent trajectory scoring. In contrast, we propose two simple approaches: 1) combining shaped, deterministic rewards with textual feedback and 2) decomposing the original problem into a sequence of sub-tasks. These approaches outperform the baselines, yielding more performant tool trajectories. These results highlight the difficulty of autonomous ML planning and point toward tool-augmented systems that rely less on subjective LLM scoring as tool sets grow in size and complexity. 1. We introduce ML-Tool-Bench, a tool-augmented benchmark for end-to-end ML planning with 61 tools and 15 Kaggle challenges. 2. We formalize scratchpad-augmented planning via named-object management that supports arbitrarily large artifacts and reversible branching in search. 3. We propose MCTS-Shaped , an MCTS approach with shaped, deterministic rewards and targeted textual feedback, which improves trajectory validity and performance over ReAct and LATS. 4. We introduce Hierarchical MCTS , an approach that decomposes problems into sequenced sub-tasks, further improving validity and robustness. For GPT-4o, Hierarchical MCTS improves over LATS by 9.93 percentile positions on the leaderboard and over ReAct by 16.52 percentile positions (median across all competitions). For GPT-4.1-mini, it improves over MCTS-Shaped by 1.89 percentile positions, while both ReAct and LATS had a median percentile position of 0. Together, these advances establish strong baselines for tool-augmented, end-to-end ML planning and reduce reliance on subjective LLM scoring. 2 Related Work Machine Learning Benchmarks for AI Agents : Most of the existing Data Science and ML benchmarks, provide the LLM agent access to write code that