TRAIL: Trace Reasoning and Agentic Issue Localization

Darshan Deshpande, Varun Gangal, Hersh Mehta, Anand Kannappan, Rebecca Qian · 2025

Read TRAIL: Trace Reasoning and Agentic Issue Localization by Darshan Deshpande, Varun Gangal, Hersh Mehta, Anand Kannappan, Rebecca Qian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08638.txt.

Introduction

Darshan Deshpande and co-authors present a 2025 paper on debugging the traces left by AI agent workflows. Agents that chain tools and reasoning steps produce long, tangled logs. Human review of those logs is slow and does not scale, so the authors set out to systematize how errors in traces are found and named.

They propose a formal taxonomy of error types seen in agent systems. With it, they built TRAIL, a set of 148 large traces labeled by human annotators. The traces come from single- and multi-agent systems doing software and information tasks. Leading long-context models were then tested at spotting the faults. Even the best model scored poorly at trace debugging. The gap shows how far automated oversight still has to go.

Catalog record

CREATORDarshan Deshpande, Varun Gangal, Hersh Mehta, Anand Kannappan, Rebecca Qian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-08638

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TRAIL: Trace Reasoning and Agentic Issue Localization Darshan Deshpande, Varun Gangal, Hersh Mehta, Anand Kannappan, Rebecca Qian Abstract The increasing adoption of agentic workflows across diverse domains brings a critical need to scalably and systematically evaluate the complex traces these systems generate. Current evaluation methods depend on manual, domain-specific human analysis of lengthy workflow traces—an approach that does not scale with the growing complexity and volume of agentic outputs. Error analysis in these settings is further complicated by the interplay of external tool outputs and language model reasoning, making it more challenging than traditional software debugging. In this work, we (1) articulate the need for robust and dynamic evaluation methods for agentic workflow traces, (2) introduce a formal taxonomy of error types encountered in agentic systems, and (3) present a set of 148 large human-annotated traces ( TRAIL ) constructed using this taxonomy and grounded in established agentic benchmarks. To ensure ecological validity, we curate traces from both single and multi-agent systems, focusing on real-world applications such as software engineering and open-world information retrieval. Our evaluations reveal that modern long context LLMs perform poorly at trace debugging, with the best gemini-2.5-pro model scoring a mere 11% on TRAIL . Our dataset and code are made publicly available to support and accelerate future research in scalable evaluation for agentic workflows 1 1 1 https://huggingface.co/datasets/PatronusAI/TRAIL . 1 Introduction Figure 1: Illustration of the TRAIL taxonomy of errors The rapid advancement of large language models (LLMs) has catalyzed the development of agentic systems capable of automating difficult, multi-step tasks across various domains such as software engineering and multi-hop IR Ma et al. (2023) ; OpenAI (2024) ; Nguyen et al. (2024) ; Wang et al. (2025a) . Unlike traditional generative models, agents can interact with diverse tools and dynamically navigate environments, often with minimal human supervision Wang et al. (2024a) . This escalation of system complexity demands more challenging and multifaceted evaluation processes Nasim (2025) and has led to the adoption of LLMs as evaluators for such agentic systems Zheng et al. (2023) ; Chen et al. (2024) ; Kim et al. (2024) ; Zhu et al. (2025) ; Deshpande et al. (2024a) . However, as multi-agent systems scale and become integral to real-world workflows, evaluating and debugging their performance remains a significant challenge. Agentic non-determinism Laban et al. (2025) ; Patronus AI (2025) and multi-step task solving Mialon et al. (2023) ; Yao et al. (2024) demand greater observability than the simple end-to-end evaluations offered by existing benchmarks Kapoor et al. (2024a) ; Zhuge et al. (2024) ; Moshkovich et al. (2025) ; Cemri et al. (2025) . Such complex environments require granular taxonomies and well-annotated traces that can serve as references for debugging and root-cause analysis of agent behaviors Cemri et al. (2025) . When creating taxonomies and benchmarks to test and improve agents, we must ensure these are grounded in real-world applications and are not centered around dummy data Bowman and Dahl (2021) ; Liu et al. (2024b) . Previous agent trace analysis frameworks have primarily focused on parsed traces containing unstructured text Cemri et al. (2025) , which do not adequately represent common agent framework outputs that generate structured traces logged in standardized formats like opentelemetry OpenTelemetry (2025) . Additionally, as observed by Guo et al. (2023) ; Sui et al. (2024) , handling structured data remains challenging for LLMs, an observation corroborated by previous research on automated software engineering trace analysis Roy et al. (2024a) ; Ma et al. (2024b) . These limitations highlight the need for new approaches specifically designed for structured agentic traces. To address these challenges and facilitate the analysis and evaluation of agentic executions, we propose a formal error taxonomy, shown in Figure 3 , that promotes granular failure diagnosis. We also present a carefully curated, turn-level annotated trace dataset called TRAIL (Trace Reasoning and Agentic Issue Localization), which demonstrates the validity and practical utility of our proposed taxonomy. In our work, we utilize and build on SWE-Bench Jimenez et al. (2024) ; Aleithan et al. (2024) and GAIA Mialon et al. (2023) while addressing three major shortcomings inherent to previous automatic agent evaluation paradigms. Firstly, we aim to replace end to end analysis of agents with a benchmark containing step-level analysis of traced agentic workflows. Secondly, we address the need for grounding in real scenarios by producing opentelemetry -based structured traces that span beyond present model context length limits. Finally, as compared to benchmarks focused only on agentic reasoning and coordination Cemri et al. (2025) ; Kokel et al. (2025) , TRAIL focuses on validity through addition of finer, more aligned system execution failures and planning error categories such as API errors and Task Orchestration Errors to our taxonomy. Such categories are not only relevant to model developers but also to users and engineers optimizing single and multi-agent AI applications. The contributions of our work are as follows: • We introduce a formal taxonomy ( Figure 1 ) that defines, fine-grained agentic error categories spanning across three key areas: reasoning, planning, and execution. • Based on this taxonomy, we present TRAIL , an ecologically grounded execution trace benchmark comprising 148 meticulously curated traces (totaling 1987 open telemetry spans, of which 575 exhibit at least one error) drawn from the GAIA Mialon et al. (2023) and SWE-Bench Jimenez et al. (2024) datasets and covering a wide range of tasks. • We show that TRAIL is a non-trivially difficult benchmark for LLMs on many fronts 1. Current SOTA LLM families such as o3 , claude-3.7-sonnet and gemini-2.5-pro perform modestly at best on TRAIL , both in terms of predicting error categories and their location. With gemini-2.5-pro the best performing model, achieving only 11% combined joint accuracy on both splits. 2. Solving TRAIL requires a significant fraction of the maximum input length of LLMs (or exceeds it), as well as requires generating significant fraction of their maximum output (See Table 2 , Figure 5 ) 3. Models benchmarked on TRAIL benefit from both the presence and greater extent of reasoning chains (§ 5.1.4 , § 5.1.5 ), highlighting the need for improvement in exploration capabilities of LLMs. • TRAIL is fully open-source (MIT License), will be accompanied by a HuggingFace leaderboard, and serves as a foundation for future research on evaluating agentic workflows. Figure 2: TRAIL trace’s span structure and error examples 2 Relevant Work LLM-as-a-Judge Shortcomings of conventional metrics such as ROUGE, BLEU, and BERTScore Schluter (2017) ; Freitag et al. (2020) ; Hanna and Bojar (2021) has led to the wide adoption of LLMs as evaluators and critics of other AI systems Zheng et al. (2023) ; Zhu et al. (2025) ; Chen et al. (2025) ; Chen et al. (2024) ; Kim et al. (2024) . Recent approaches have enhanced LLM judges’ reasoning capabilities through techniques like unconstrained evaluation plan and specialized training methods that enable more robust evaluation performance across diverse scenarios Lightman et al. (2023) ; Wang et al. (2024e) ; Trivedi et al. (2024) ; Saha et al. (2025) . The evaluation landscape has evolved significantly with the introduction of frameworks like FLASK Ye et al. (2024b) which decompose coarse-level scoring into skill set-level evaluations for each instruction, demonstrating high

Read the complete text