ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents

Kenan Li, Qirui Jin, Liao Zhu, Xiaosong Huang, Yijia Wu, Yikai Zhang, Xin Zhang, Zijian Jin, Yufan Huang, Elsie Nallipogu, Chaoyun Zhang, Yu Kang, Saravan Rajmohan, Qingwei Lin, Wenke Lee, Dongmei Zhang · 2026

Read ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents by Kenan Li, Qirui Jin, Liao Zhu, Xiaosong Huang, Yijia Wu, Yikai Zhang, Xin Zhang, Zijian Jin, Yufan Huang, Elsie Nallipogu, Chaoyun Zhang, Yu Kang, Saravan Rajmohan, Qingwei Lin, Wenke Lee, Dongmei Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07789.txt.

Introduction

Li and 15 colleagues measure how much different information signals help software-engineering agents succeed. Recent agent workflows first gather intermediate signals during exploration, such as reproduction tests, regression tests, edit locations, execution context, and API usage. No earlier work systematically compares these signals or measures how far agent-found signals lag behind oracle signals. This paper quantifies what oracle signals and agent-extracted signals each contribute to final task success. It also maps the remaining headroom in each signal, showing where better exploration would pay most.

Catalog record

CREATORKenan Li, Qirui Jin, Liao Zhu, Xiaosong Huang, Yijia Wu, Yikai Zhang, Xin Zhang, Zijian Jin, Yufan Huang, Elsie Nallipogu, Chaoyun Zhang, Yu Kang, Saravan Rajmohan, Qingwei Lin, Wenke Lee, Dongmei Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88086
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.07789

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents Kenan Li, Qirui Jin, Liao Zhu, Xiaosong Huang, Yijia Wu, Yikai Zhang, Xin Zhang, Zijian Jin, Yufan Huang, Elsie Nallipogu, Chaoyun Zhang, Yu Kang, Saravan Rajmohan, Qingwei Lin, Wenke Lee, Dongmei Zhang Abstract Recent advances in language model (LM) agents have significantly improved automated software engineering (SWE). Prior work has proposed various agentic workflows and training strategies as well as analyzed failure modes of agentic systems on SWE tasks, focusing on several contextual information signals: Reproduction Test , Regression Test , Edit Location , Execution Context , and API Usage . However, the individual contribution of each signal to overall success remains underexplored, particularly their ideal contribution when intermediate information is perfectly obtained. To address this gap, we introduce Oracle-SWE, a unified method to isolate and extract oracle information signals from SWE benchmarks and quantify the impact of each signal on agent performance. To further validate the pattern, we evaluate the performance gain of signals extracted by strong LMs when provided to a base agent, approximating real-world task-resolution settings. These evaluations aim to guide research prioritization for autonomous coding systems. 1 Introduction As language models (LMs) demonstrate increasingly sophisticated programming capabilities, the demand for rigorous evaluation frameworks has grown. While early datasets like Defects4J ( Just et al. 2014 ) and LiveCodeBench ( Jain et al. 2024 ) test standalone code generation, the evaluation landscape has decisively shifted toward repository-level problem solving. In SWE-bench ( Jimenez et al. 2023 ) , models are required to function as software engineering agents: they must interact with an executable environment, examine the entire repository, comprehend its organization, and address tasks originating from real-world GitHub issues. The difficulty and practical relevance of SWE-bench have led to extensive follow-up benchmarks— including SWE-bench-Multilingual ( Yang et al. 2025b ) , SWE-bench-Multimodal ( Yang et al. 2025a ) , SWE-bench-Live ( Zhang et al. 2025 ; Li et al. 2026 ) , and SWE-bench-Pro ( Deng et al. 2025 ) —each further increasing task diversity and complexity. To tackle the tasks defined in SWE benchmarks, researchers have developed a diverse set of approaches ranging from fully agentic frameworks to simpler agent-free pipelines. Systems such as Yang et al. 2024 ; Wang et al. 2024 ; Xia et al. 2024 enable LMs to operate over real code repositories by organizing the problem-solving process into structured stages. Subsequent work has further specialized this process by training models that focus on particular stages of the workflow ( Ma et al. 2025b ; Wei et al. 2025 ) , while other studies ( Meng et al. 2024 ; Bouzenia & Pradel 2025 ) analyze execution traces of agents to better understand common failure patterns and potential improvement points. From these prior efforts, we observe that most of them implicitly rely on a small set of critical contextual information signals (or sources / factors, synonymous in the following context) when solving SWE tasks, which can be categorized into five major categories: Reproduction Test, Regression Test, Edit Location, Execution Context, and API Usage (Table 1 ). Though numerous works focus on the five signals, one key question remains: how much could these systems improve potentially if specific information signals were made perfectly accurate? The question is critical in guiding future research priorities. To answer this, we construct oracle versions of these signals and measure their individual impact through controlled ablations. Experimental results reveals that for SWE-bench, the contribution order is Reproduction Test > Execution Context \sim Edit Location > API Usage > Regression Test , while for SWE-bench-Live and Pro, the order becomes Reproduction Test > Execution Context \sim API Usage > Edit Location > Regression Test . The strong effect of reproduction suggests that current LM agents often struggle to resolve ambiguous issue descriptions and benefit greatly from explicit failure signals produced by high-quality tests. Since such oracle information is not available in real issue resolution, we further conduct an additional validation experiment by prompting a stronger LM agent to find the five signals respectively and evaluate their contribution with the original agent. The result is consistent with the oracle version. The contributions of this work are threefold: • Identifying the critical information factors in SWE benchmarks. This work formally defines five critical information factors previous works focus on through comprehensive literature review and proposes a method to extract ground truths of the 5 factors. The extraction method is generalized to multiple SWE benchmarks to benefit dataset analysis. • Quantifying upper bounds through oracle analysis. This work estimate the potential contribution upper bounds of the five signals to overall task success, guiding research priorities. • Agent-based validation of oracle insights. This work further designs a validation experiment to estimate the contribution of the five factors in real-world issue resolution. 2 Related Works 2.1 Agentic Workflow in SWE Tasks A substantial body of work on SWE benchmarks focuses on designing agent pipelines that structure repository exploration into a sequence of stages supported by tool usage. For example, Agentless ( Xia et al. 2024 ) adopts a canonical staged procedure in which an LM first performs bug localization, then generates code edits, constructs reproduction tests, and finally validates the resulting patch using both reproduction and regression tests. Similarly, SWE-agent ( Yang et al. 2024 ) and OpenHands ( Wang et al. 2024 ) follows a multi-stage instruction of localization, reproduction, editing and test validation, with a minimal tool set of shell commands and string-replacement editor to interact with the repository. Closed-source systems such as Claude Code ( Anthropic 2025 ) follow a comparable paradigm but introduce advanced tools such as planning tool ToDoWrite and delegation tool Task that delegates sub-tasks to sub-agents. Empirical analysis by ( Bouzenia & Pradel 2025 ) further shows that despite architectural differences, agent trajectories across benchmarks such as Defects4J and SWE-bench tend to exhibit three recurring behavioral phases: Localization (explore, search, and identify relevant code regions), Edit (reason about the bug and produce a patch), and Test Execution (reproduce the bug and verify correctness through tests) (Figure 7 (a)). Another line of work explicitly decomposes SWE tasks into multi-agent or multi-task pipelines with specialized components. TDFlow ( Han et al. 2025 ) distributes the workflow across agents dedicated to localization, patch generation, format correction, reproduction-test synthesis, and test-based validation. Several multi-agent approaches further introduce a specialized agent for code-graph construction to improve contextual understanding and identify relevant API utilities. MarsCode ( Liu et al. 2024 ) and Lingma Agent ( Ma et al. 2025a ) build code graphs that combine repository structure, abstract syntax trees, and inter-function call relations, improving localization, contextual reasoning, and API-usage understanding; GraphLocator ( Liu et al. 2025 ) augments these graphs with natural-language descriptions to facilitate navigation. AutoCodeRecover ( Zhang et al. 2024 ) and PatchPilot ( Li et al. 2025 ) obtain code graphs of function call relations by executing reproduction tests and collecting execution traces. 2.2 Training LMs for SWE Tasks In addition to improving agentic systems, another research direction focuses on directly training language models to perform key sub-tasks in the SWE pipeline, including issue localization, patch synthesis, and reproduction-test generation. For instance, SoRFT ( Ma et al. 2025b )

Read the complete text