TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software maintenance tools

Minxing Wang, Xiaofei Xie, Yintong Huo · 2026

Read TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software maintenance tools by Minxing Wang, Xiaofei Xie, Yintong Huo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26563.txt.

Introduction

Minxing Wang, Xiaofei Xie, and Yintong Huo diagnose coding failures with TrajAudit. Repository agents leave long, noisy trajectories: steps blur failure causes, and low-signal observations muddy reasoning. It pairs an investigator with modules. One folds noisy context off-target through semantic saliency folding. The other derives diagnostic guidance from test failure reports, used as prior knowledge. RootSE holds 102 cases with earliest error steps annotated. It beats baselines by 10.8 and 21.6 percent in localization accuracy, with and without references.

Catalog record

CREATORMinxing Wang, Xiaofei Xie, Yintong Huo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52812
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software maintenance tools Minxing Wang, Xiaofei Xie, Yintong Huo Abstract . Agentic systems have been widely studied to automate software engineering jobs such as bug fixing. As these systems increasingly tackle complex tasks, understanding where and why they fail becomes essential for iterative refinement and operational reliability. Existing automated failure diagnosis approaches leverage task execution trajectories , yet their effectiveness degrades substantially as trajectory length and complexity increase. For repository-level coding tasks specifically, trajectories are laden with noise, such as redundant program structure and verbose code context. Moreover, these trajectories are very long, while long-context reasoning remains a known weakness of LLMs. To address these two challenges, we propose TrajAudit , the first failure diagnosis framework for repository-level coding trajectories. TrajAudit employs an investigator agent supported by two modules: one filters failure-irrelevant information through pattern matching and keyword detection, and the other generates a preliminary diagnosis from test failure reports as prior knowledge, helping the agent handle noisy long contexts. The investigator agent can further invoke tools to retrieve filtered content on demand, ensuring that critical information is preserved while noise is minimized. We also introduce RootSE , a benchmark of 93 real-world agentic failure instances sourced from software maintenance tasks, representing the most complex trajectory diagnosis benchmark to date. Experiments on RootSE show that TrajAudit outperforms all existing baselines by over 24.4 percentage points in localization accuracy, while reducing token consumption by at least 18%, demonstrating its practical effectiveness. We hope this work draws community attention to failure management in agentic software engineering and provides a foundational resource for future research. 1. Introduction LLM-based agentic systems are autonomous systems powered by large language models (LLMs) that perceive environmental states, perform goal-oriented reasoning, and execute actions in a closed-loop manner ( 3 ; 14 ; 33 ) . Recently, a growing number of LLM-based agents have been developed to automate repository-level software maintenance tasks , like issue resolution and feature development ( 20 ; 58 ; 47 ; 22 ; 41 ) . While these agents succeed in simple tasks such as single-file modifications, they still struggle with complex, multi-file tasks that require long-horizon reasoning and execution ( 17 ; 13 ; 53 ; 31 ) . These failures occur in opaque ways, often as the cumulative consequence of a single early mistake, such as a misunderstanding of the task requirements or a flawed implementation plan. Therefore, understanding where and why agents fail is critical for the iterative refinement of agentic systems and, ultimately, trustworthy intelligent software engineering ( 40 ; 5 ; 21 ; 32 ) . Execution trajectory , which records sequential steps of the agent’s reasoning, tool invocations, and environmental observations, provides the key information to monitor the agent’s behavior ( 39 ; 56 ) . As shown in Figure 2 , each step contains four types of information: the thinking process, the response, the actions taken, and the resulting observations. These trajectories are commonly used for understanding failures behind agent execution. For example, one pioneering study conducted in-depth manual investigations of inter-step inconsistencies to characterize failure patterns, such as reasoning-action conflicts ( 6 ) . To further automate failure diagnosis , Zhang et al. ( 57 ) proposed three failure localization methods that feed trajectory content into LLMs all at once, step-by-step, or via binary search to identify the decisive failure step. In addition, inspired by spectrum-based fault localization (SBFL) used in traditional software engineering, Ge et al. ( 15 ) proposed FAMAS, which identifies failure-suspicious steps by comparing multiple trajectories from the same task and flagging the most frequently occurring steps ( 24 ; 1 ; 51 ) . Figure 1. Failure diagnosis in agentic systems. However, these approaches can only handle simple trajectories with few execution steps, such as the one from web browsing ( 57 ; 34 ) . Their diagnostic performance drops below 40% when applied to long-horizon task trajectories (often exceeding 40 steps) in repository-level coding problems. In particular, we identify two distinct challenges as follows, (1) Observational noise. Observations refer to the information returned by tools invoked by the agent, often accounting for over 70% of the total trajectory content. However, most observations are not relevant to failure localization, such as redundant program structures and verbose code context, which can interfere with LLM reasoning ( 44 ) . (2) Excessive length. These trajectories often span from 20 to over 100 steps, with each step containing on average over 5,000 characters. Existing studies ( 7 ; 45 ; 30 ) show that even state-of-the-art LLMs struggle to maintain reasoning quality when processing long contexts. Figure 2. The agent workflow and execution trajectory in a coding task. Both challenges stem from a fundamental limitation of existing diagnosis methods: they passively consume entire trajectories, treating all steps as equally relevant. Inspired by the action-taking nature of agents ( 42 ; 43 ) , we propose an agentic approach that actively explores and fetches the most relevant fragments from long and noisy trajectories. To this end, we introduce TrajAudit , the first automated framework for agent trajectory failure diagnosis, including pinpointing the first step at which the agent takes an action that introduces an error (i.e., earliest decisive error step ) and providing diagnosis justifications. TrajAudit addresses the aforementioned challenges through an agent assisted by two synergistic modules. (1) Prior failure reasoning derives a preliminary diagnosis by prompting an LLM to identify the most suspicious region responsible for the failure based on the failed test code and its corresponding error description. This diagnosis is then incorporated into the localization agent’s context as prior knowledge, directing the model’s focus toward the most probable failure segments, effectively mitigating long-context degradation. (2) Semantic saliency folding selectively compresses trajectory observations by retaining only failure-relevant context, such as code patch structures and entries containing failure indicators (e.g., ’fail’, ’exception’). (3) Investigator agent enables dynamic access to the full trajectory, performing on-demand retrieval of folded content through predefined interactive APIs. This mechanism enables a top-down diagnostic approach, allowing the diagnosis to begin with a high-level overview and selectively drill down into details on demand, thereby mitigating observational noise. Furthermore, to evaluate TrajAudit, we introduce RootSE , the first benchmark to evaluate a model’s ability to diagnose agent execution failure in completing software engineering jobs. The dataset consists of 93 complex instances with over 4,500 execution steps, offering a comprehensive testbed for identifying the earliest decisive error points. Experimental results on RootSE demonstrate that TrajAudit outperforms the strongest baseline by 24.4 percentage points in localization accuracy, while consuming 18% fewer tokens. In summary, the main contributions of this paper are threefold: • Framework. We introduce TrajAudit, the first automated failure diagnosis framework to localize the error step and offer diagnosis justification in agent coding trajectories. • Benchmark. We curate RootSE, a novel benchmark comprising 93 complex instances and 4,500 execution steps, to evaluate the model’s ability in diagnosing agent failure. • Evaluation. We evaluate existing methods and TrajAudit on RootSE, demonstrating that our approach significantly

Read the complete text