Scalable Supervising Software Agents with Patch Reasoner
Junjielong Xu, Boyin Tan, Xiaoyuan Liu, Chao Peng, Pengfei Gao, Pinjia He · 2025
Read Scalable Supervising Software Agents with Patch Reasoner by Junjielong Xu, Boyin Tan, Xiaoyuan Liu, Chao Peng, Pengfei Gao, Pinjia He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22775.txt.
Introduction
Testing code patches is costly, and good tests are rare. Junjielong Xu leads a team with five co-authors. In 2025 they propose R4P, a patch reasoning method. It gives rewards for agent training without running tests. That removes heavy, fragile test sandboxes.
R4P compares patches in a group against each other. Changes in each patch reveal likely quality. This yields a dense reward signal. It works across agent scaffolds. On SWE-bench patch checks, accuracy proved competitive. The team also built a test free scaffold called Mini-SE. Training it with R4P raised its pass rate a lot. Test time patch choice raised it further.
Catalog record
| CREATOR | Junjielong Xu, Boyin Tan, Xiaoyuan Liu, Chao Peng, Pengfei Gao, Pinjia He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22775
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scalable Supervising Software Agents with Patch Reasoner Junjielong Xu, Boyin Tan, Xiaoyuan Liu, Chao Peng, Pengfei Gao, Pinjia He Abstract While large language model agents have advanced software engineering tasks, the unscalable nature of existing test-based supervision is limiting the potential improvement of data scaling. The reason is twofold: (1) building and running test sandbox is rather heavy and fragile , and (2) data with high-coverage tests is naturally rare and threatened by test hacking via edge cases. In this paper, we propose R4P, a patch verifier model to provide scalable rewards for training and testing SWE agents via reasoning. We consider that patch verification is fundamentally a reasoning task, mirroring how human repository maintainers review patches without writing and running new reproduction tests. To obtain sufficient reference and reduce the risk of reward hacking, R4P uses a group-wise objective for RL training, enabling it to verify multiple patches against each other’s modification and gain a dense reward for stable training. R4P achieves 72.2% Acc. for verifying patches from SWE-bench-verified, surpassing OpenAI o3. To demonstrate R4P’s practicality, we design and train a lite scaffold, Mini-SE, with pure reinforcement learning where all rewards are derived from R4P. As a result, Mini-SE achieves 26.2% Pass@1 on SWE-bench-verified, showing a 10.0% improvement over the original Qwen3-32B. This can be further improved to 32.8% with R4P for test-time scaling. Furthermore, R4P verifies patches within a second, 50x faster than testing on average. The stable scaling curves of rewards and accuracy along with high efficiency reflect R4P’s practicality. 1 Introduction Large language models (LLMs) agents have made notable progress in software engineering (SWE) thanks to rule-based reinforcement learning (RL) and test-time scaling (TTS) ( Jimenez et al. 2023 ; Luo et al. 2025a ) . However, unlike tasks with extensive publicly available and easy-to-check answers (e.g., math), SWE tasks naturally have non-unique answers (patches) that are hard to formally verify ( ter Beek et al. 2024 ) . Thus, testing was adopted as a proxy approach ( Tihanyi et al. 2025 ) for verifying patches in SWE’s RL and TTS. Yet, testing is heavy , hindering data scaling in SWE. Specifically, building test environment (e.g., Docker images) for each data instance is highly labor-intensive ( Pan et al. 2024 ; Jain et al. 2025 ) , and maintaining sandbox instances (e.g., Docker containers) for parallel testing is unstable due to surge workload ( Luo et al. 2025a ) . Furthermore, the inherent test coverage problem ( Yu et al. 2025 ) also harms the reliability of test results, making tests being easily hacked by edge cases when expanding data to less maintained GitHub projects. We found that only 28.11% of resolved issues have corresponding tests on projects with over 500 stars ( Appendix A ). This hinders the utilization of vast, unlabeled data from the open-source community. We identify the core challenge is how to acquire test-free supervision for patches at scale . Ideally, a reward model ( Minghao Yang 2024 ; Shiwen et al. 2024 ; Chen et al. 2025 ) could serve as an alternative patch verifier. However, they are trained to judge the relative preference of solutions than their absolute correctness. Since SWE is a hard reasoning task, all LLM rollout answers may be incorrect and should not be rewarded, making such relative assessment inadequate. Furthermore, without test execution, for an LLM to reliably identify subtle incompatibilities, it needs an exhaustive, agentic search across the repository’s call graph and dependencies. However, this approach makes the verification process as hard and slow as the patch generation process itself ( Meng et al. 2024 ) , making it impractical for scalable supervision. To mitigate this, some test-time verifiers ( Pan et al. 2024 ; Jain et al. 2025 ; Luo et al. 2025a ) use certain agent’s trajectories as an additional input, trained to maximize the log probability of token YES or NO from a process-oriented perspective. However, such task formulation creates a strong coupling between the model and a specific interaction style, hindering its ability to generalize to other agents. In this paper, we introduce R4P , a reasoning patch verifier model for SWE agents that do not rely on any golden test, developer patch, agent trajectory, or run-time sandbox. It enables efficient data scaling for real-world issues that are untested or have low test coverage, offering potential for continual learning beyond the constraints of test-based sandbox data. We consider patch verification is fundamentally a reasoning task which can be improved via RL. This mirrors how real-world repository maintainers evaluate pull request (PR) patches based on their understanding of the repository and a detailed analysis of the patch, rather than relying on writing and running test scripts to judge correctness ( Baum et al. 2016 ) . To avoid reward hacking and enable effective test-free verification, R4P adopts a group-wise training objective. Specifically, for a given issue and a set of candidate patches, R4P assesses each patch by comparing it against others in the group for mutual contextual information, compensating for the absence of tests and facilitating the detection of subtle errors. Moreover, by expanding the verifier’s solution space beyond binary classification, R4P reduces the risk of reward hacking and offers a denser reward signal compared to the original sparse bipolar reward. This leads to more stable training and improved convergence. We evaluate R4P on patch generated by four different agents from SWE-bench-verified Experiments. When built upon Qwen2.5-Coder-32B-Instruct, R4P achieves 72.2% accuracy, surpassing advanced models like OpenAI o3. To further demonstrate R4P’s practicality, we developed Mini-SE , a lite, execution-free agentic scaffold with issue-resolving-oriented code search and edit capabilities. We then use R4P for supervision and train Mini-SE based on Qwen3-32B model on R2E-Gym issues without using its sandbox. While Mini-SE does not test its generated patches during rollout, its verification burden is transferred to R4P for patch selection for test-time scaling. The Pass@1 resolution rate on SWE-bench-verified steadily improves with more training data and finally reaches 26.2% , outperforming Lingma Agent + Lingma SWE-GPT-72B ( Ma et al. 2024 ) . With R4P for patch selection, it can be further improved to 32.8% . In addition, R4P verifies each patch within a second on average, 50x faster than the minute-level time cost of testing. These show the practicality of reasoning-based verification. 1 1 1 The repository is released at GitHub Figure 1: In this paper, we explore a reasoning-based patch verification strategy to provide scalable supervision for software engineering agents. This approach (1) mitigates data scarcity caused by test quality requirements in open-source codebases, (2) removes the need for environment setup and makes data expansion costless, and (3) greatly reduces computational overhead compared to heavy test execution. We aim to leverage such imperfect yet easily scalable supervision to enhance model capability even after high-quality test data is exhausted. 2 Preliminaries Reinforcement Learning : Reinforcement learning (RL) aims to learn an optimal policy \pi_{\theta} that maximizes the expected cumulative discounted reward r when interacting with an environment. In the context of auto-regressive LLM, state at step t is the concatenation of prompt x and current response y_{<t} , and the action is the t-th token y_{t} . Emprically, policy gradient methods are widely