Reinforcement Learning for Machine Learning Engineering Agents

Sherry Yang, Joy He-Yueya, Percy Liang · 2025

Read Reinforcement Learning for Machine Learning Engineering Agents by Sherry Yang, Joy He-Yueya, Percy Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01684.txt.

Introduction

Sherry Yang, Joy He-Yueya, and Percy Liang show that small AI agents can learn to do machine learning work better by practice. Most such agents just prompt a large fixed model, so they never improve with use. The team trains a much smaller model with reinforcement learning, where rewards guide updates after each try. Two snags arise: tasks take uneven time, and test scores give blunt feedback.

The authors fix the time snag by weighting slow but good tries more in updates. For blunt feedback, a helper model adds log lines to code so partial progress can earn partial credit. On a bench of Kaggle tasks, the small trained agent beat a large prompted model by a clear margin. The result favors learning from practice over sheer model size.

Catalog record

CREATORSherry Yang, Joy He-Yueya, Percy Liang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01684

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reinforcement Learning for Machine Learning Engineering Agents Sherry Yang, Joy He-Yueya, Percy Liang Abstract Existing agents for solving tasks such as ML engineering rely on prompting powerful language models. As a result, these agents do not improve with more experience. In this paper, we show that agents backed by weaker models that improve via reinforcement learning (RL) can outperform agents backed by much larger, but static models. We identify two major challenges with RL in this setting. First, actions can take a variable amount of time (e.g., executing code for different solutions), which leads to asynchronous policy gradient updates that favor faster but suboptimal solutions. To tackle variable-duration actions, we propose duration-aware gradient updates in a distributed asynchronous RL framework to amplify high-cost but high-reward actions. Second, using only test split performance as a reward provides limited feedback. A program that’s nearly correct is treated the same as one that fails entirely. To address this, we propose environment instrumentation to offer partial credit , distinguishing almost-correct programs from those that fail early (e.g., during data loading). Environment instrumentation uses a separate static language model to insert print statement to an existing program to log the agent’s experimental progress, from which partial credit can be extracted as reward signals for learning. Our experimental results on MLEBench suggest that performing gradient updates on a much smaller model (Qwen2.5-3B) trained with RL outperforms prompting a much larger model (Claude-3.5-Sonnet) with agent scaffolds, by an average of 22% across 12 Kaggle tasks. 1 Introduction Language model (LM) agents using external tools can perform complex tasks from writing software programs ( jimenez2023swe ; yang2024swe ) to conducting scientific research ( baek2024researchagent ) . Perhaps eventually, LMs can make better versions of themselves through agents performing machine learning engineering (MLE) ( huang2023mlagentbench ; chan2024mle ) . Existing agents for MLE rely on simply prompting powerful LMs. While scaling up test-time compute ( snell2024scaling ; wu2024inference ) with prompting alone can allow an agent to find better solutions, the agent’s behavior does not change drastically without gradient updates despite much more experiences being collected. As shown in Figure 1 , running the best MLE prompting framework according to MLEBench chan2024mle for days leads to only slightly better best solutions. Figure 1 : Performing gradient update with RL on Qwen2.5-3B ( blue ) is more effective in improving the best task performance than prompting Claude3.5-Sonnet with the best agent scaffold ( red ) on the “leaf-classification” task from MLEBench ( chan2024mle ) . A natural approach to improving performance given past experience is to perform gradient updates using reinforcement learning (RL) ( sutton1998reinforcement ) . However, agentic settings pose additional challenges to RL. First, action execution of an MLE agent can take a variable amount of time (e.g., training different ML models), which leads to asynchronous policy gradient updates that favor faster but suboptimal solutions. To overcome this challenge, we propose duration-aware gradient updates during distributed asynchronous RL training to balance gradient updates to faster and slower actions. With duration-aware updates, we observe that an agent stops favoring faster solutions, achieving better performance in the long run. Second, while performance on the test split serves as a natural choice of rewards, they offer limited feedback, treating a nearly correct program (e.g., which failed during the last step of writing the solution to the correct location) the same as one that fails entirely (e.g., during data loading). Such limited feedback may lead to the agent stuck in suboptimal solutions (e.g., using the easiest way to produce test labels). To address limited feedback, we propose environment instrumentation to offer partial credit to intermediate steps of completing an ML task (e.g., loading the data, building and training a model). We implement environment instrumentation by using a static copy of the original LM to insert print statements in the code generated by the agent, the execution of which provides partial credit. We observe that partial credit can gradually guide the agent away from making trivial mistakes (e.g., import errors, failures to load data) and towards improving ML techniques (e.g., feature engineering and hyperparameter choices). Across a set of 12 Kaggle tasks from MLEBench ( chan2024mle ) , we show that RL training of a much smaller open-weight model (Qwen2.5-3B ( yang2024qwen2 ) ) can eventually outperform prompting much larger close-weight models (e.g., Claude-3.5-Sonnet and GPT-4o) by an average of 22% and 24%, respectively. Our results suggest that future LM agent should learn to balance the compute resources spent across inference, interactions (action execution), and performing gradient updates, for tasks whose action execution incur non-trivial overhead, as in ML engineering. 2 Background In this section, we provide relevant notations and define key learning objectives. We further discuss a few challenges of running standard RL algorithms in agentic settings. MLE Agent in a Markov Decision Process (MDP). We consider a Markov Decision Process (MDP) ( puterman2014markov ) represented by a tuple \langle\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\mu\rangle , consisting of a state space \mathcal{S} , an action space \mathcal{A} , a reward function \mathcal{R}:\mathcal{S}\times\mathcal{A}\to\mathbb{R} , a state transition probability function \mathcal{P}:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) , and an initial state distribution \mu\in\Delta(\mathcal{S}) . A policy \pi:\mathcal{S}\to\Delta(\mathcal{A}) interacts with the environment, starting from an initial state s_{0}\sim\mu . At each interactive step k\geq 0 , an action a_{k}\sim\pi(s_{k}) is sampled from the policy and applied to the environment. The environment then transitions into the next state s_{k+1}\sim\mathcal{P}(\cdot|s_{k},a_{k}) while a scalar reward \mathcal{R}(s_{k},a_{k}) is produced. Reinforcement Learning (RL) aims to find a policy \pi that maximizes the expected future rewards: J(\pi)=E_{\pi,\mu,\mathcal{P}}\left[\sum_{k=0}^{K}\mathcal{R}(s_{k},a_{k})\right], (1) where K is the total number of steps. Standard RL algorithms such as policy gradient ( williams1992simple ; schulman2017proximal ) can be applied to learn the policy update rule by estimating \nabla J(\pi_{\theta})=E_{\pi,\mu,\mathcal{P}}\left[\sum_{k=0}^{K}\nabla_{\theta}\log\pi_{\theta}(a_{k}|s_{k})\hat{A}(s_{k},a_{k})\right], (2) where \hat{A}(s_{k},a_{k}) is some advantage function which can be separately estimated (e.g., by Monte-Carlo returns from \pi ( williams1992simple ) ). In the MLE agent setting, \mathcal{S} captures input to the agent, including problem description, datasets, and any experiment history. \mathcal{A} captures solutions generated by an agent, including high-level plans (e.g., which family of models to use) and low-level code. \mathcal{P} captures any potential output from the environment during action execution (e.g., error messages). We focus on the K=1 setting where the agent tries to solve the problem in one generation. \mathcal{R} captures rewards from the environment, such as performance on the test split or any additional scalar-value feedback the environment is able to offer to the agent. Challenges of RL for MLE Agents. For sample efficiency ( thrun1992efficient ; kakade2003sample ) , many RL training frameworks implement an asynchronous distributed setup where multiple “actors” can interact with their own instances of the environment simultaneously, gathering experiences which are then sent to a “learner” for policy gradient updates ( liang2018rllib ; hoffman2020acme ) . In agentic settings such as ML engineering, each action may take a variable amount of time to execute. As a result, running distributed RL training favors faster actions (slower actions might often time out). Moreover, time-consuming actions are sampled less frequently in a distributed training framework, leading to an uneven number of gradient updates for faster and slower actions. As shown in Figure 3 , naïvely running a distributed RL framework ( sheng2024hybridflow ) on

Read the complete text