SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling

Haoran Wang 1∗ , Zhenyu Hou 1∗ , Yao Wei 2 , Jie Tang 1 , Yuxiao Dong 1 1 Tsinghua University 2 Zhipu AI Code Models & Data · 2025

Read SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling by Haoran Wang 1∗ , Zhenyu Hou 1∗ , Yao Wei 2 , Jie Tang 1 , Yuxiao Dong 1 1 Tsinghua University 2 Zhipu AI Code Models & Data (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07636.txt.

Introduction

Software engineering agents that fix code on their own need good training runs and fair tests, and both are scarce. Haoran Wang and four co-authors built such an agent, SWE-Dev, in 2025 on open language models, with pipelines that manufacture the missing material.

One pipeline synthesizes test cases to grade proposed patches, and another scales up recorded agent trajectories for training. On the SWE-bench Verified benchmark, their seven-billion-parameter model solved about 23 percent of tasks and the thirty-two-billion model about 37 percent. Both topped the open-source rivals in the comparison. Code, models, and data are released in public, so other teams can build on the recipe.

Catalog record

CREATORHaoran Wang 1∗ , Zhenyu Hou 1∗ , Yao Wei 2 , Jie Tang 1 , Yuxiao Dong 1 1 Tsinghua University 2 Zhipu AI Code Models & Data
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07636

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling Haoran Wang 1∗ , Zhenyu Hou 1∗ , Yao Wei 2 , Jie Tang 1 , Yuxiao Dong 1 1 Tsinghua University 2 Zhipu AI Code Models & Data Abstract Large language models (LLMs) have advanced rapidly from conversational problem solving to addressing real-world tasks involving tool use, such as software engineering (SWE). Recent LLM-powered SWE systems, such as OpenAI Codex and Cursor, have offered end-to-end automation of the software development process. However, building effective SWE agents remains challenging due to the lack of high-quality training data and reliable test-time evaluation. To address this issue, we present SWE-Dev , an SWE agent built upon open-source LLMs, with a focus on training and inference scaling. For training scaling, we develop a robust pipeline to synthesize test cases and scale up agent trajectories to construct the training data. For inference scaling, we increase the interaction budget within a single run to enable further thinking within one independent attempt. Experiments on the SWE-bench-Verified benchmark show that the SWE-Dev models can achieve top performance among all open SWE agents. Specifically, the resolve rate of our 7B and 32B models reach 23.4% and 36.6%, respectively, outperforming state-of-the-art open-source models. All code, models, and datasets are publicly available at https://github.com/THUDM/SWE-Dev . 1 1 footnotetext: Work partially done when interned at Zhipu AI. 1 Introduction Large language models (LLMs) have rapidly evolved from generating simple code snippets to tackling more complex tasks, such as competitive programming ( Li et al. 2022 ; OpenAI 2025 ) , machine learning problems ( Chan et al. 2024 ) , and real-world software engineering (SWE) tasks ( Xi et al. 2024 ; Jimenez et al. 2024 ; Zan et al. 2025 ; Lyu et al. 2023 ) . Among these tasks, SWE is particularly hard and challenging ( OpenAI 2025 ; Cursor 2024 ) , but highly useful for improving real-world productivity. Unlike simple code generation, SWE tasks usually require LLMs to interact with complex and fragile runtime environments, solve toolchain issues, execute scripts, and reason over large, interdependent codebases ( Ma et al. 2024b ) . Figure 1: The SWE-Dev performance with training and inference scaling. Notably, SWE-Dev-32B achieves a resolve rate of 34.0%, matching the performance of GPT-4o even without inference scaling. The SWE tasks are usually evaluated on benchmarks such as SWE-bench ( Jimenez et al. 2024 ) and its recent multimodal extensions ( Yang et al. 2024b ; Zan et al. 2025 ) . These benchmarks require models to generate verifiable, test-passing solutions on real-world codebases. To achieve this, the models must be capable of step-by-step reasoning, tool using, and long-horizon planning. To date, training models to handle these tasks requires reliable reward signals, typically derived from test cases that validate the correctness of the solutions. However, most existing datasets lack such test cases or executable environments ( Chen et al. 2021 ; Lyu et al. 2024 ) , making it difficult to evaluate solutions or provide useful feedback during training. This deficiency limits models’ abilities to iteratively refine their outputs through trial-and-error, thus constraining their potential to solve real-world SWE tasks that need grounded, verifiable solutions. To address this issue, we introduce SWE-Dev , an open-source SWE agent framework coupled with a scalable test case generation pipeline. This pipeline works in two-stages: First, the LLMs are used to generate Gherkin-style descriptions—a structured natural language format used to specify test scenarios. Second, a code generator outputs code patches for further validation. Empirical analysis shows that these synthesized test cases align closely with the original problem semantics. Through large-scale experiments, we identify a clear training scaling trend: increasing the number of sampled agent trajectories leads to improved downstream performance. To enhance efficiency, we propose an LLM-based filtering method that selects high-quality trajectories. This allows us to maintain the benefits of a full dataset while retaining only the most valuable data. We also propose iteration scaling—a simple yet effective strategy that aims to scale up inference budget by increasing the number of interaction rounds during a single evaluation episode. This reduces the need for repeated re-evaluation, making it particularly advantageous in scenarios where access to test oracles is expensive or delayed. In addition, we explore advanced post-training strategies, including Rejection Sampling Fine-Tuning (RFT) ( Yuan et al. 2023 ) , KTO ( Ethayarajh et al. 2024 ) , and OREO ( Wang et al. 2024a ) . We observe that RFT brings the most significant performance improvement, while offline reinforcement learning (RL) methods—KTO and OREO—deliver marginal or task-specific gains. We build SWE-Dev based on the open Qwen2.5-Coder ( Hui et al. 2024 ) , Llama-3.1 ( Llama 2024 ) , and GLM4-9B ( GLM 2024 ) models. Its performance is evaluated on the SWE-bench-Verified benchmark, with results shown in Figure 1 . With the Qwen2.5-Coder-32B model, SWE-Dev achieves a resolve rate of 36.6% on SWE-Bench-Verified, representing state-of-the-art performance among open-source SWE agents. 1. Test Case Generation Pipeline. We build a scalable LLM-based pipeline to generate real-world SWE instances and their executable test cases. Through this pipeline, we successfully construct 2,000 test cases by filtering 38,000 high-quality issues across 4,000 repositories. 2. Scaling Trends in Data and Inference. We empirically identify scaling trends between training data volume, number of interaction steps, and model performance. We find that the SWE-Dev-32B model improves from 34.0% to 36.6% resolve rate by adding 45 more interaction turns, highlighting the benefit of multi-step execution for agents. 3. Post-Training Recipe for SWE Agents. We examine several post-training methods, including RFT, KTO, and OREO, as well as hybrid combinations. RFT consistently outperforms others by effectively leveraging high-quality samples, demonstrating its robustness and scalability for training SWE agents. Figure 2: Pipeline for test case generation, divided into description generation and code generation phases. The pipeline begins with extracting repository information, followed by generating Gherkin scenarios and then detailed test cases. An optional revision step leverages traceback errors to refine the generated test cases. The final output includes fail-to-pass test cases. 2 Related Works SWE Dataset Construction. Prior benchmarks like SWE-bench ( Jimenez et al. 2024 ) , DevEval ( Li et al. 2024b ) , EvoCodeBench ( Li et al. 2024a ) and Commit0 ( Zhao et al. 2024 ) crawl from a large dataset and manually annotate the desired instance, which is labor consuming. There are works trying to filter existing fail-to-pass test cases like SWE-Gym ( Pan et al. 2024 ) , which annotates 2.4k instances by hand. Filtering out from a large dataset ( Golubev et al. 2024 ) is effective but will leave out many useful issues only because the corresponding PR does not contain test cases. Therefore, automatically generating test cases from issue descriptions becomes essential. SWE Agentic Framework. SWE agent framework focuses on two mainstream types: Interaction-based frameworks and pipeline-based frameworks. Interaction-based frameworks like OpenHands ( Wang et al. 2024c ) , SWE-Agent ( Yang et al. 2024a ) , Learn-By-Interact ( Su et al. 2025 ) and Wang et al. 2024b usually pre-define a set of agent-computer interfaces (ACIs) to help model manipulate the environment. Meanwhile, pipeline-based frameworks will design the whole process into several steps, like Agentless ( Xia et

Read the complete text