MLE-Dojo : Interactive Environments for Empowering LLM Agents in Machine Learning Engineering

Rushi Qiang , Yuchen Zhuang , Yinghao Li, Dingu Sagar V K, Rongzhi Zhang, Changhao Li, Ian Shu-Hei Wong, Sherry Yang §, Percy Liang §, Chao Zhang, Bo Dai · 2025

Read MLE-Dojo : Interactive Environments for Empowering LLM Agents in Machine Learning Engineering by Rushi Qiang , Yuchen Zhuang , Yinghao Li, Dingu Sagar V K, Rongzhi Zhang, Changhao Li, Ian Shu-Hei Wong, Sherry Yang §, Percy Liang §, Chao Zhang, Bo Dai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07782.txt.

Introduction

Rushi Qiang and co-authors build an interactive playground for training AI agents at machine learning engineering in a 2025 paper. Static benchmarks test one-shot answers, but real engineering is a loop of trying, debugging, and refining. Their framework lets agents work through that loop with feedback at each step.

The setup draws on more than two hundred real data-science contests, covering data cleaning, model search, tuning, and code repair. Agents can be trained by fine-tuning or reinforcement learning inside the fully runnable environment. Tests of eight leading models showed real gains across iterations, yet also clear limits in long projects and hard debugging. The team open-source the system to spur community work.

Catalog record

CREATORRushi Qiang , Yuchen Zhuang , Yinghao Li, Dingu Sagar V K, Rongzhi Zhang, Changhao Li, Ian Shu-Hei Wong, Sherry Yang §, Percy Liang §, Chao Zhang, Bo Dai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07782

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MLE-Dojo : Interactive Environments for Empowering LLM Agents in Machine Learning Engineering Rushi Qiang , Yuchen Zhuang , Yinghao Li, Dingu Sagar V K, Rongzhi Zhang, Changhao Li, Ian Shu-Hei Wong, Sherry Yang §, Percy Liang §, Chao Zhang, Bo Dai Abstract We introduce MLE-Dojo , a Gym-style framework for systematically reinforcement learning, evaluating, and improving autonomous large language model (LLM) agents in iterative machine learning engineering (MLE) workflows. Unlike existing benchmarks that primarily rely on static datasets or single-attempt evaluations, MLE-Dojo provides an interactive environment enabling agents to iteratively experiment, debug, and refine solutions through structured feedback loops. Built upon 200+ real-world Kaggle challenges. MLE-Dojo covers diverse, open-ended MLE tasks carefully curated to reflect realistic engineering scenarios such as data processing, architecture search, hyperparameter tuning, and code debugging. Its fully executable environment supports comprehensive agent training via both supervised fine-tuning and reinforcement learning, facilitating iterative experimentation, realistic data sampling, and real-time outcome verification. Extensive evaluations of eight frontier LLMs reveal that while current models achieve meaningful iterative improvements, they still exhibit significant limitations in autonomously generating long-horizon solutions and efficiently resolving complex errors. Furthermore, MLE-Dojo ’s flexible and extensible architecture seamlessly integrates diverse data sources, tools, and evaluation protocols, uniquely enabling model-based agent tuning and promoting interoperability, scalability, and reproducibility. We open-source our framework and benchmarks to foster community-driven innovation towards next-generation MLE agents: https://github.com/MLE-Dojo/MLE-Dojo . 1 1 1 MLE-Dojo webpage and leaderboard available at https://mle-dojo.github.io/MLE-Dojo-page/ and https://huggingface.co/spaces/MLE-Dojo/Leaderboard Figure 1: Benchmark evaluations of eight frontiers LLMs across 50 evaluation tasks in MLE-Dojo . 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities across diverse coding tasks, including code generation, debugging, and refactoring [ 3 , 14 , 1 , 23 ] . Despite these advances, machine learning engineering (MLE) tasks remain uniquely challenging due to their inherent complexity, specialized domain knowledge, and the extensive iterative experimentation required for developing and optimizing ML algorithms [ 17 , 26 , 2 ] . LLM-based agents hold significant promise for revolutionizing MLE by automating repetitive steps, generating boilerplate code, suggesting suitable algorithms, debugging implementations, and iteratively refining models. Ultimately, advanced LLM-based MLE agents could autonomously perform tasks such as literature search, hypothesis generation, experimental design, method implementation, result analysis, and even scientific dissemination. However, developing robust and autonomous MLE agents remains in its infancy, largely due to the absence of comprehensive benchmarks, interactive and executable training environments, and standardized evaluation frameworks capable of supporting rigorous, iterative experimentation. Several recent benchmarks have emerged to assess and facilitate progress in LLM-based coding agents [ 22 , 34 , 37 , 17 , 21 , 26 , 29 ] . However, most existing efforts focus mainly on isolated tasks such as data analysis or visualization, or competitions with narrow, non-interactive scenarios [ 22 , 34 , 37 ] . Such compartmentalized benchmarks fail to capture the inherent complexity and iterative nature of real-world MLE workflows, which require continuous experimentation, iterative debugging, structured feedback incorporation, and efficient resource management. Although broader benchmarks such as MLE-Bench [ 17 ] and DSBench [ 21 ] provide more diverse MLE tasks, they still lack interactive environments that support iterative experimentation and training paradigms such as fine-tuning or reinforcement learning. Moreover, in contrast to structured software engineering (SWE) tasks [ 20 , 35 ] , which only require a structured environment for code-based problem-solving and can be readily sourced from general web resources such as GitHub issues, typical MLE tasks necessitate systematically curated datasets and standardized training data, elements currently unavailable in existing MLE benchmarks or gym-style frameworks [ 2 , 26 ] . This complexity not only increases storage and computational requirements, but also creates significant challenges when scaling to more comprehensive and diverse problem sets. In this study, we introduce MLE-Dojo , an interactive, large-scale Gym-style environment explicitly designed for developing and benchmarking autonomous LLM agents on real-world MLE workflows. Built upon 200+ real-world Kaggle competitions across critical ML domains, including tabular data analysis, computer vision, natural language processing, and time series forecasting, etc. . MLE-Dojo provides carefully curated, executable tasks that closely replicate real-world engineering challenges. Among these, 150 tasks constitute the initial training dataset for MLE agents and are integrated into an interactive environment, enabling training trajectory sampling for both supervised fine-tuning and reinforcement learning. Moreover, MLE-Dojo incorporates pre-installed dependencies, standardized evaluation scripts, and real-time outcome verification, thus simplifying iterative experimentation and debugging processes. We conduct extensive empirical evaluations involving eight LLMs and multiple agent scaffolds. All evaluation results are publicly accessible via a continuously updated leaderboard, promoting transparent comparison, reproducibility, and collaborative research in this emerging field. Additionally, MLE-Dojo features a modular architecture that decouples agent capabilities from the underlying environment, facilitating seamless integration with diverse tools and data sources, enhancing interoperability, scalability, and supporting a robust ecosystem for developing generalizable MLE agents. Our main contributions are summarized as follows: • Comprehensive Framework and Benchmark : We establish MLE-Dojo as a comprehensive and large-scale benchmark consisting of over 200 Kaggle MLE competitions, enabling systematic and rigorous evaluations of autonomous LLM agents. • Interactive and Executable Environment : MLE-Dojo provides an interactive and fully executable Gym-style environment that facilitates iterative experimentation, including comprehensive training trajectories sampling for supervised fine-tuning and reinforcement learning. • Advanced Functionalities and Scalability Supports : MLE-Dojo uniquely facilitates outcome verification, model-agnostic agent tuning, and seamless integration of diverse datasets and tools, significantly accelerating the development of robust, generalizable, and scalable MLE agents. • Extensive Empirical Evaluation and Public Leaderboard : We conduct large-scale evaluations across multiple state-of-the-art LLMs and agent scaffolds, with results publicly available through an actively maintained long-term real-time leaderboard to foster community-driven innovation. In the remainder of the paper, we discuss the related work in section 2 , detail our data curation process in section 3 , and thoroughly describe MLE-Dojo in section 4 . We present the experimental setup and results of LLMs as MLE agents in section 5 and conclude the paper in section 6 . 2 Related Works Datasets Interactive Gym Executable Environment Training Facilities Extensible Tasks Flexible Scaffolds Tabular CV NLP # Training # Eval AutoKaggle [ 24 ] ✗ ✗ ✗ ✗ ✗ ✔ ✗ ✗ 0 8 MLAgentBench [ 17 ] ✗ ✗ ✗ ✗ ✗ ✔ ✔ ✔ 0 13 DSBench [ 21 ] ✗ ✗ ✗ ✗ ✗ ✔ ✔ ✔ 0 74 DSEval [ 36 ] ✗ ✔ ✗ ✗ ✗ ✔ ✔ ✔ 0 31 MLEBench [ 2 ] ✗ ✔ ✗ ✗ ✔ ✔ ✔ ✔ 0 75 MLGym [ 26 ] ✔ ✔ ✗ ✗ ✗ ✔ ✔ ✔ 0 13 MLE-Dojo ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 150 50 Table 1: Summary of existing benchmarks of MLE agents with data resources and sample sizes. X -Agent (Coding). LLM-based coding agents have significantly advanced automated software development, with impactful applications across two primary domains: SWE and MLE. SWE agents [ 33 , 8 , 32 ] excel at code generation, debugging, and refactoring for improved maintainability and performance. For example, SWE-agent [ 33 ] operates within a constrained agent-computer interface to facilitate file creation, repository navigation, and code testing; Magentic-One [ 8 ] expands the capability of SWE agents with web navigation features, further

Read the complete text