Training Software Engineering Agents and Verifiers with SWE-Gym

Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, Yizhe Zhang · 2024

Read Training Software Engineering Agents and Verifiers with SWE-Gym by Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, Yizhe Zhang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-21139.txt.

Description

A 2024 arXiv research paper by Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, Yizhe Zhang. We present SWE-Gym, the first environment for training software engineering (SWE) agents. SWE-Gym contains 2,438 real-world task instances, each comprising a Python codebase with an executable runtime environment, unit tests, and a task specified in natural language. We use SWE-Gym to train language model based SWE agents, and achieve up to 19% absolute gains in resolution rate on the popular SWE-Bench Verified and Lite test sets.

Catalog record

CREATORJiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, Yizhe Zhang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-21139

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training Software Engineering Agents and Verifiers with SWE-Gym Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, Yizhe Zhang Abstract We present SWE-Gym, the first environment for training software engineering (SWE) agents. SWE-Gym contains 2,438 real-world task instances, each comprising a Python codebase with an executable runtime environment, unit tests, and a task specified in natural language. We use SWE-Gym to train language model based SWE agents, and achieve up to 19% absolute gains in resolution rate on the popular SWE-Bench Verified and Lite test sets. We also experiment with inference-time scaling through verifiers trained on agent trajectories sampled from SWE-Gym. When combined with our fine-tuned SWE agents, we achieve 32.0% and 26.0% on SWE-Bench Verified and Lite, respectively, reflecting a new state-of-the-art for open-weight SWE agents. To facilitate further research, we publicly release SWE-Gym, models, and agent trajectories. † † affiliationnotice: * Equal contribution. {\dagger} Equal supervision. 1 Introduction Figure 1 : SWE-Gym enables scalable improvements for software engineering agents. Top : Scaling the amount of training data shows consistent performance improvements as we obtain more training trajectories, with no signs of saturation at 491 trajectories. We use temperature t=0 for evaluation. Bottom : For inference time scaling, we generate a number of candidate trajectories per task and select the best using a verifier trained on SWE-Gym. This approach demonstrates roughly log-linear gains with the number of sampled solutions. t=0 (excluded from regression) is used as the first hypothesis to be consistent with the top figure; later rollouts use t=0.5 . Language models (LMs) have remarkable promise in automating software engineering (SWE) tasks, as most clearly measured by recent progress on benchmarks like SWE-Bench ( Jimenez et al., 2024 ) and Commit0 ( Zhao et al., 2024 ) . While LM-based SWE agents have shown significant performance gains through improving agent-computer interfaces ( Yang et al., 2024 ) and prompting strategies ( Wang et al., 2024c ) , advances in SWE agents have been limited by a reliance on proprietary models, with limited research to improve the underlying LM itself. Unlike other domains where supervised fine-tuning and reinforcement learning have significantly improved LM capabilities, such as chat ( Ouyang et al., 2022 ) , math reasoning ( Shao et al., 2024 ; Yuan et al., 2024 ) , and web navigation ( Pan et al., 2024 ) , software engineering currently lacks suitable training environments, and creating environments is uniquely challenging. Real-world software engineering requires interaction with an executable runtime that has been prepared with the appropriate software dependencies and reproducible test suites, among other requirements. These challenges are reflected in the existing resources (Tab. 1 ). For example, the SWE-Bench ( Jimenez et al., 2024 ) training split contains only solutions (git patches that solve the task), missing the step-by-step actions taken by the developer to create each solution, and executable environments and reward signals. R2E ( Jain et al., 2024 ) uses synthetic tasks that are very far from real-world problems, while datasets such as APPS ( Hendrycks et al., 2021a ) focus only on isolated tasks rather than realistic repository-level coding problems. Table 1 : SWE-Gym is the first publicly available training environment combining real-world SWE tasks from GitHub issues with pre-installed dependencies and executable test verification. Repository-level : whether each task is situated in a sophisticated repository; Executable Environment : whether each task instance comes with an executable environment with all relevant dependencies pre-installed; Real task : whether task instruction is collected from human developers. Dataset (split) Repository-Level Executable Environment Real task # Instances (total) # Instances (train) CodeFeedback ( Zheng et al., 2024b ) ✗ ✗ ✓ 66,383 66,383 APPS ( Hendrycks et al., 2021a ) ✗ ✓ ✓ 10,000 5,000 HumanEval ( Chen et al., 2021 ) ✗ ✓ ✓ 164 0 MBPP ( Tao et al., 2024 ) ✗ ✓ ✓ 974 374 R2E ( Jain et al., 2024 ) ✓ ✓ ✗ 246 0 SWE-Bench (train) ( Jimenez et al., 2024 ) ✓ ✗ ✓ 19,008 19,008 SWE-Gym Raw ✓ ✗ ✓ 64,689 64,689 SWE-Bench (test) ( Jimenez et al., 2024 ) ✓ ✓ ✓ 2,294 0 SWE-Gym ✓ ✓ ✓ 2,438 2,438 To bridge this gap, we present SWE-Gym, the first training environment combining real-world software engineering tasks from GitHub issues with pre-installed dependencies and executable test verification. SWE-Gym contains 2,438 Python tasks sourced from 11 popular open-source repositories (Tab. ), providing useful environments for training LMs as agents and verifiers. SWE-Gym supports training state-of-the-art open-weight SWE agents . Based on the OpenHands ( Wang et al., 2024c ) agent scaffold for general-purpose software development (§ 2 ), we fine-tune a 32B Qwen-2.5 coder model ( Hui et al., 2024b ) using only 491 agent-environment interaction trajectories sampled using SWE-Gym, and achieve substantial absolute improvements of +12.3% (to 15.3%) and +13.6% (to 20.6%) in resolution rate on SWE-Bench Lite and SWE-Bench Verified respectively (§ 4.2 ). SWE-Gym is effective across agent scaffolds . In another agent scaffold based on a specialized workflow (MoatlessTools; Örwall 2024 ; § 2 ), we experiment with self-improvement, where the LM interacts with SWE-Gym, receives reward from it, and learns to improve itself through rejection sampling fine-tuning. This self-improvement boosts performance up to 19.7% on SWE-Bench Lite. SWE-Gym supports training verifier models to enable inference-time scaling . We use test suites included in SWE-Gym to determine whether sampled agent trajectories are successful or not. Given these samples, we train a verifier model ( Cobbe et al., 2021 , i.e., an outcome-supervised reward model; ) that estimates a trajectory’s probability of success. This enables inference-time scaling, where we sample multiple agent trajectories and select the one with the highest estimated reward according to the verifier. This further improves the resolution rate to 32.0% (+11.4% absolute improvement) on SWE-Bench Verified (§ 5.1.1 ; Fig. 1 bottom) and 26.0% on SWE-Bench Lite (§ 5.1.2 ), establishing a new state-of-the-art among systems with publicly accessible weights (Tab. 9 ). Our baseline training and inference-time scaling methods on SWE-Gym yield continuously improved results with increasing compute (Fig. 1 ). In the training phase, performance scales with the number of sampled trajectories up to our current limit of 491 trajectories, suggesting that performance is currently limited by the compute budget for sampling rather than the number of tasks in SWE-Gym. Similarly, using the agent and verifier trained by SWE-Gym, the bottom panel shows that using more compute during inference time steadily improves the performance. 2 Related Work Agents that solve GitHub issues. We focus on software engineering agents designed to automatically resolve GitHub issues within the SWE-Bench framework ( Jimenez et al., 2024 ) . These agents take a GitHub issue and its associated code repository as input and generate a valid code modification (i.e., a git diff patch) to address the issue. The correctness of these modifications is verified using a human-written test suite. Existing agent designs are categorized by the extent of human priors integrated into their workflows: Specialized workflows ( Xia et al., 2024 ; Örwall, 2024 ; Zhang et al., 2024b ; Chen et al., 2024 ) involve human-defined stages (e.g., localization, code editing, patch re-ranking), where a LM is iteratively prompted for each

Read the complete text