From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level

Jia Li, Yuxin Su, Michael R. Lyu · 2026

Read From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level by Jia Li, Yuxin Su, Michael R. Lyu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03731.txt.

Introduction

Jia Li, Yuxin Su, and Michael R. Lyu present RepoReason, a diagnostic benchmark for how well AI agents reason across whole code repositories. Real projects are massive webs of interdependent files, and an agent must stay logically consistent across all of them. Existing tests swing between tiny snippets and black-box scores that hide where reasoning breaks.

RepoReason centers on verifying assertions about program behavior, using an execution-driven mutation framework. Mutations regenerate ground truth from the running environment itself, which blocks memorized answers while keeping authentic logic depth. A fine-grained system slices programs to measure reading load, simulation depth, and integration width separately. Tests of frontier models revealed a common aggregation deficit, where models handle pieces yet fail to combine them, a diagnosis vendors can act on.

Catalog record

CREATORJia Li, Yuxin Su, Michael R. Lyu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03731

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level Jia Li, Yuxin Su, Michael R. Lyu Abstract As large language models (LLMs) evolve into autonomous agents, evaluating Repository-Level Reasoning , the ability to maintain logical consistency across massive, interdependent file systems, has become critical. Current benchmarks typically fluctuate between isolated code snippets and black-box evaluations. We present RepoReason , a white-box diagnostic benchmark centered on Abductive Assertion Verification . To eliminate memorization while preserving authentic logical depth, we implement an Execution-Driven Mutation framework that utilizes the environment as a Semantic Oracle to regenerate ground-truth states. Furthermore, we establish a fine-grained diagnostic system using dynamic program slicing, quantifying reasoning via three orthogonal metrics: ESV (Reading Load), MCL (Simulation Depth), and DFI (Integration Width). Comprehensive evaluations of frontier models (e.g., Claude-4.5-Sonnet, DeepSeek-v3.1-Terminus) reveal a prevalent Aggregation Deficit , where integration width serves as the primary cognitive bottleneck. Our findings provide granular white-box insights for optimizing the next generation of agentic software engineering. 1 Introduction The paradigm of software engineering is undergoing a fundamental shift: from human-centric coding to agentic autonomous development ( 15 ; 13 ) . As Large Language Models ( LLMs ) evolve into autonomous software agents, the core competency required has escalated from performing localized execution reasoning within isolated code snippets—akin to controlled laboratory exercises—to maintaining logical consistency across extensive, interdependent file systems—a capability we define as Repository-Level Reasoning. This capability imposes profound challenges on LLMs , demanding not only code synthesis but also the agency to explore vast contexts to pinpoint target definitions, aggregate multi-source information across complex dependencies, and perform long-chain reasoning to accurately track and compute state mutations. However, a critical gap exists between this growing demand and current evaluation methods. Existing benchmarks largely fall into two extremes. On one side, function-level reasoning datasets, such as CRUXEval ( 12 ) and REval ( 5 ) , study code execution reasoning within isolated snippets. While these provide analytical rigor, they operate in a laboratory setting that lacks the cross-file dependencies and complex architectural contexts found in real-world software engineering. On the other side, frameworks like SWE-bench ( 16 ) evaluate agents in wild repository environments but function as black-boxes: they check whether a problem is solved but offer little insight into why an agent fails. Without detailed diagnosis, the path to improving agent reasoning remains unclear. To address this gap, we introduce RepoReason , a repository-level code reasoning benchmark designed as a white-box diagnostic tool. Unlike traditional generation tasks, RepoReason shifts to a verification-centric approach: “Given the complex execution history of this repository, what is the current system state?” By requiring models to derive deterministic values that satisfy assertions rather than writing implementation logic, we separate core logical reasoning from syntactic noise ( e.g. , formatting errors, indentation slips, or API name hallucinations). This ensures that every failure reflects a genuine deficit in reasoning rather than superficial coding errors. To ensure the benchmark reflects real-world complexity, we extract task instances from a curated set of mainstream Python repositories ( e.g. , toolz ( 10 ) , sympy ( 27 ) , jinja2 ( 21 ) ), challenging models with authentic, deep logic. Beyond static code extraction, we perform concrete execution of the repositories’ built-in unit test suites to capture granular runtime traces. By designing structural metrics based on these traces—such as stack depth, function call density, and cross-file dependency breadth—we quantitatively filter and retain only the most complex reasoning tasks. Constructing such a benchmark requires navigating between two distinct failure modes of current evaluations: Spurious Correlation (guessing via shallow patterns) and Rote Memorization (recalling via pre-training data). To address this, we propose a two-stage defense mechanism. First, to prevent shortcut learning via visual pattern matching, we reject raw trace values and instead identify Unit Test Assertions as our semantic anchors. While raw traces are often localized and easily inferred from immediate code patterns, assertions function as semantic milestones that encapsulate high-level developer intent, strategically positioned to validate the system state after complex logical operations. They force Abductive Reasoning ( 17 ) , requiring the model to reconstruct the preceding execution history across complex call stacks to deduce the verified state. However, a significant obstacle remains: many assertions within these mainstream repositories’ unit tests have already been encountered by LLMs during pre-training. This leads to a risk of data leakage ( 8 ) . To resolve this, we implement an Execution-Driven Mutation methodology. We treat the code execution environment as a Semantic Oracle. By keeping the reasoning logic (the call graph filtered via structural metrics) intact but perturbing the program inputs, we effectively sever the model’s memory retrieval path. We then re-execute the entire repository context to capture the new ground-truth state, subsequently masking these verified values to create cloze-style reasoning tasks. Finally, to ensure these captured states are objectively evaluable, we introduce a Deterministic Value Protocol, which filters runtime data to eliminate representational drift and ensure stable, unique ground-truth values. This design ensures that while the reasoning logic remains authentic and complex, the specific state values are unmemorized and deterministic. Finally, through an extensive evaluation of a diverse range of frontier models including Claude-Sonnet-4.5 and DeepSeek-v3.1-Terminus, we demonstrate that even top-tier agents face severe performance degradation when confronted with high-entropy logic. Our results unveil a critical Aggregation Deficit where accuracy drops sharply as integration width increases, as well as a Cliff Effect in reading comprehension beyond 600 lines and a significant loss of consistency beyond 100 execution steps. These insights highlight that next-generation agents must transcend local reasoning and focus on high-breadth information synthesis and long-chain state consistency. In summary, this paper makes the following contributions: • From Prediction to Abductive Verification: We pioneer a repository-level reasoning paradigm that pivots from direct outcome prediction to abductive assertion verification. By centering tasks on masked assertions within interdependent file systems, we successfully decouple core architectural intelligence from surface-level syntactic noise. • The Semantic Oracle Framework: We resolve the Richness-Contamination Dilemma through an Execution-Driven Mutation framework. By utilizing the execution environment as a Semantic Oracle, we sever data leakage paths via input perturbation and real-time state regeneration, ensuring tasks remain logically authentic yet unmemorized. • White-box Diagnostic Instrument: Beyond end-to-end scoring, we introduce a granular diagnostic system powered by dynamic program slicing. We formalize three orthogonal cognitive metrics (Reading Load, ESV ; Simulation Depth, MCL ; Integration Width, DFI ) that map reasoning failures to specific cognitive bottlenecks: context overload, state tracking deficits, and aggregation barriers. • Empirical Unveiling of Aggregation Deficits: Through extensive evaluation of frontier models, we provide the first quantitative analysis of agentic reasoning trajectories across massive codebases (up to 776k+ LoC). Our findings identify DFI as the primary bottleneck for next-generation agents, empirical confirming a severe aggregation deficit in information synthesis. Table 1: Comparison of RepoReason with existing code reasoning and understanding benchmarks. Benchmark Repo-Lvl Scale (LoC) Evaluation Paradigm Metric Granularity Diagnostic Depth HumanEval ( 6 ) \times \sim 10 Code Generation Pass/Fail Black-box MBPP ( 4 ) \times \sim 10 Code Generation Pass/Fail Black-box ClassEval ( 11 ) \times \sim 40 Code Generation Pass/Fail Black-box CRUXEval ( 12 ) \times \sim 10 Execution Prediction Pass/Fail Black-box REval ( 5 ) \times

Read the complete text