Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

Akshay Manglik, Apaar Shanker, Kaustubh Deshpande, Jason Qin, Yash Maurya, Veronica Chatrath, Vijay S. Kalmath, Levi Lentz, Yuan (Emily) Xue · 2026

Read Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents by Akshay Manglik, Apaar Shanker, Kaustubh Deshpande, Jason Qin, Yash Maurya, Veronica Chatrath, Vijay S. Kalmath, Levi Lentz, Yuan (Emily) Xue (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21347.txt.

Introduction

Akshay Manglik and co-authors present Insights Generator, a system for corpus-level trace diagnostics of language-model agents. Hand diagnosis, inspecting a few traces and forming ad hoc hypotheses, misses patterns visible only across trace populations and fails to scale to production corpora. Insights Generator formalizes corpus-level diagnostics: given a trace corpus, produce grounded insights on systematic patterns across trace groups, each linked to evidence. A multi-agent system proposes and tests hypotheses across the corpus to build an evidence-backed report.

Catalog record

CREATORAkshay Manglik, Apaar Shanker, Kaustubh Deshpande, Jason Qin, Yash Maurya, Veronica Chatrath, Vijay S. Kalmath, Levi Lentz, Yuan (Emily) Xue
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85196
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21347

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents Akshay Manglik, Apaar Shanker, Kaustubh Deshpande, Jason Qin, Yash Maurya, Veronica Chatrath, Vijay S. Kalmath, Levi Lentz, Yuan (Emily) Xue Abstract Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG) , a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4 pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG’s scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality. \contact [email protected] † † footnotetext: † Work conducted while at Scale AI. † † footnotetext: ∗ Corresponding author: [email protected] 1 Introduction Agentic systems produce long execution traces of reasoning steps, tool calls, and environment interactions. Debugging is typically driven by evaluation results that flag failed runs; practitioners then inspect a small subset of traces and form hypotheses [ 7 , 15 ] . Because evaluations are defined over final task outcomes, they yield aggregate signals but do not reveal intermediate behaviors. Additionally, they do not surface patterns across the broader space of scenarios. Many errors, inefficiencies, and performance differences consequently remain difficult to identify [ 7 , 11 ] . This is especially true for failures localized to specific subsets of inputs, task types, or execution contexts that are not individually prominent enough to noticeably shift aggregate metrics. Such patterns are invisible to standard evaluation unless the practitioner already knows to look for them. Many of these signals emerge only at the cohort level. Systematic differences in tool usage, recurring reasoning structures, and patterns correlated with strong or weak performance become visible only when traces are analyzed collectively. Cohort comparison is a particularly powerful form of this analysis: examining groups of traces that vary along a single controlled axis, such as model variant, prompt version, or task category, isolates the behavioral differences attributable to that axis and provides a stronger targeted diagnostic signal than aggregate metrics alone. Beyond diagnosis of known failure modes, there is a distinct need for open-ended discovery . Silent failures, behavioral trends that reduce reliability without triggering explicit error signals, are a key example; they rarely appear in evaluation dashboards and require population-level analysis to detect. We refer to such findings as trace insights : grounded natural-language statements characterizing salient patterns across groups of traces, each supported by trace-level evidence. For example, a representative IG insight might read: 84% of incorrect Python-using traces exhibit ‘silent computation failures’: the code executes without errors but implements the wrong mathematical model. Cross-checking fails to catch these because it verifies internal consistency within the wrong framework rather than challenging the framework itself. Observed in 94/112 incorrect Python-using traces (37.6% of all traces); 0/19 cross-checking traces detected the modeling error. Surfacing trace insights at corpus scale presents two challenges. Scale and Complexity: Trace corpora routinely exceed any single model context window. Individual traces interleave thousands of tokens of reasoning, tool calls, and environment state. The patterns of interest, i.e. the silent failures, systematic inefficiencies, behavioral differences between cohorts are semantic and not reducible to keyword matching or predefined taxonomies [ 3 , 13 ] . Prior work addresses fragments of this problem through single-trajectory debugging [ 27 , 10 , 2 , 7 ] , classification under fixed failure taxonomies [ 4 , 11 , 24 ] , outcome-based categorization [ 16 ] , or scaffold-centric analysis that treats traces as secondary [ 9 ] , but none performs iterative pattern discovery directly over a corpus of traces. No Evaluation Standard: Trace insights are free-form claims about population-level behavior; whether such a claim is correct, grounded in cited evidence, and useful to a downstream practitioner cannot be reduced to a single existing metric, and no shared methodology has been proposed. We present Insights Generator (IG) , a system that addresses the first challenge through iterative hypothesis-driven aggregation over trace corpora: it decomposes an open-ended diagnostic question into testable hypotheses, validates each at corpus scale, and synthesizes evidence-backed findings. We make the following contributions: • Iterative corpus-level trace analysis. IG separates hypothesis generation from corpus-scale validation and routes all trace access and processing through a stateful Python layer, enabling a decompose-hypothesize-validate loop that operates over corpora exceeding any single context window. This surfaces patterns, such as the silent computation failures above, that single-pass and per-trajectory methods miss (Section 2 ). • Evaluation framework for insights. We introduce a four-setting framework that varies who evaluates (calibrated LLM judge vs. human expert) and what is measured (report quality vs. downstream scaffold impact), and apply it to a range of trace-analysis systems on multiple benchmarks. Rankings are consistent across the four settings, providing the first systematic basis for comparing corpus-level trace diagnostic systems (Section 4 ). • Measurable practitioner impact. Subject matter experts using IG reports improve scaffold performance by 30.4 pp over the unmodified baseline, nearly doubling the 16.2 pp gain from the next-best analysis system. This gap is consistent with IG’s higher scores on evidence depth and mechanism explanation in the automated evaluation, and establishes that diagnostic quality translates directly to practitioner effectiveness (Section 4.2 ). Figure 1 : Insights Generator (IG) system overview. Left: the input layer provides a diagnostic question, Q , trace corpus, \mathcal{C} , and processed data store, \mathcal{S} . Center: the Orchestrator dispatches Scout agents ( \mathcal{H} : hypothesize over sampled traces) and Investigator agents ( \mathcal{H}^{*} : validate via corpus-scale cohort comparison). The Investigator analyzes \mathcal{H}^{*} to generate findings, \mathcal{F}_{r} , which are sent to the orchestrator. The orchestrator then synthesizes and de-duplicates \mathcal{F}_{r} to generate the final report. Right: the output is an evidence-backed report with findings, fixes, citations, and prevalence estimates. Bottom: the shared tool layer. Algorithm 1 formalizes the analysis loop. 2 Related Work Single-Trajectory Error Analysis. Most existing agent debugging work focuses on individual trajectory analysis. Many such methods use counterfactual replay to isolate failure causes: AgentDebug [ 27 ] reasons about alternative decision points, AgenTracer [ 22 ] injects programmatic faults and re-executes, and DoVer [ 10 ] replays trajectories with targeted interventions to validate failure hypotheses. AgentRX [ 2 ] draws on software verification techniques, inferring structured constraints from tool schemas and checking whether agent actions satisfy them. A complementary line classifies known failure types under predefined taxonomies (MAST [ 4 ] , AgentFail [ 11 ] , Who&When [ 24 ] ), but such approaches cannot surface failure modes the taxonomy did not anticipate. Even

Read the complete text