BenchTrace: A Bench mark for T esting R eflection A bility and C ontrolled E volution in LLM Agents

Jiahao Huang, Fei Cheng, Junfeng Jiang, Zefan Yu, Akiko Aizawa, [2pt] University of Tokyo · 2026

Read BenchTrace: A Bench mark for T esting R eflection A bility and C ontrolled E volution in LLM Agents by Jiahao Huang, Fei Cheng, Junfeng Jiang, Zefan Yu, Akiko Aizawa, [2pt] University of Tokyo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29225.txt.

Introduction

Jiahao Huang et al. present BenchTrace, testing reflection and controlled evolution in agents. Self-growing agents improve by looking back on misses, but testing counts only task scores and relies on agents' own runs, without set misses. BenchTrace uses 1,821 episodes across six tasks, with reflection tests on finding misses and growth tests on dodging. A failure avoidance rate metric counts the share dodging the failure. Qwen3-32B and GPT-4.1 fall below 30 percent on reflection, with diagnosis the bottleneck, and evolution lifts dodging but suffers loss and bad transfer.

Catalog record

CREATORJiahao Huang, Fei Cheng, Junfeng Jiang, Zefan Yu, Akiko Aizawa, [2pt] University of Tokyo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56737
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29225

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BenchTrace: A Bench mark for T esting R eflection A bility and C ontrolled E volution in LLM Agents Jiahao Huang, Fei Cheng, Junfeng Jiang, Zefan Yu, Akiko Aizawa, [2pt] University of Tokyo Abstract Self-evolving agents improve over time by reflecting on past failures, but existing evaluation is limited in two ways: it measures only task scores, leaving reflection quality unknown, and it relies on agents’ own episode runs, offering no mechanism to target specific failure patterns. We present BenchTrace , a benchmark for evaluating self-evolution ability in LLM agents. BenchTrace is built on a snapshot-reflection dataset of 1,821 annotated episodes spanning six diverse tasks, and comprises a Reflection Evaluation that probes failure identification through targeted QA tasks, and an Evolution Evaluation that tests whether past failure experience translates into avoidance behavior in a controlled self-evolution simulation. Building on BenchTrace, we propose failure avoidance rate (FAR) , a new evaluation metric measuring the fraction of test cases in which the agent successfully avoids the target failure instance. Experiments with Qwen3-32B and GPT-4.1 reveal that both models fall below a 30% end-to-end pass rate on reflection evaluation, with diagnosis as the primary bottleneck. Evolution evaluation shows that self-evolution methods generally improve FAR over the non-evolving baseline, but agents forget early lessons as noise episodes accumulate, and agents fail to generalize their reflections beyond the specific context, causing negative transfer across task contexts. Our correlation analysis further reveals that only a fully correct reflection is strongly associated with higher FAR. BenchTrace exposes concrete limits of current self-evolution approaches and provides a controlled, model-agnostic framework for targeted evaluation. Figure 1: (a) Traditional evaluation of self-evolving agents measures only the final task score. (b) BenchTrace constructs its dataset through three stages: Snapshot Collection, Failure Detection, and Reflection Annotation. (c) BenchTrace comprises a Snapshot-Reflection Dataset and an Evaluation Suite with a Reflection Evaluation and an Evolution Evaluation . 1 Introduction Large language model (LLM) agents have demonstrated remarkable capabilities across a wide range of complex, long-horizon agentic tasks ( 19 ) . A particularly promising research direction is self-evolution : unlike traditional LLMs that remain static after deployment, a self-evolving agent continuously improves its performance by learning from past experience and failures ( 12 ) . Self-evolving agents improve iteratively across episodes ( 4 ) , where each episode is a complete, self-contained interaction trajectory between an agent and a task environment (see Appendix A for examples). Self-evolving agents typically improve through non-parametric methods, such as accumulating failure lessons in external memory ( 12 ; 16 ; 21 ) , iteratively refining the agent’s guiding prompt ( 8 ) , and building reusable skill artifacts ( 18 ) . The self-evolution performance of an agent ultimately depends on two factors: the base LLM ’s ability to reflect on past failures, and the agent ’s evolution algorithm and its effectiveness. However, existing evaluation of self-evolving agents suffers from two issues (Figure 1 (a)). First, the interpretability issue : it measures only whether task scores improve, without decoupling the base LLM’s reflection ability from the agent’s evolution effectiveness. Second, the controllability issue : since evaluation relies on the agent’s own episode runs, the failure experiences it encounters cannot be controlled, making it impossible to isolate specific aspects of self-evolution. We therefore present BenchTrace , a Bench mark for T esting R eflection A bility and C ontrolled E volution in LLM agents. BenchTrace is built on a snapshot-reflection dataset. A snapshot is an agent-generated episode, recording the full interaction trajectory and final outcome. A reflection is a structured annotation of the core failure instances in the snapshot, organized along the detection, localization, and diagnosis hierarchy. Detection asks whether the snapshot contains failures worth addressing. Localization identifies in which steps the failure occurred. Diagnosis classifies the failure mode and explains what caused it. Figure 1 (c) shows a representative example from the dataset. BenchTrace addresses the interpretability issue by decoupling the base LLM’s reflection ability from the agent’s evolution effectiveness through two independent evaluations. The Reflection Evaluation directly probes whether the base LLM can identify failures in past episodes through QA tasks at each hierarchical level, isolating reflection quality from task outcomes. The Evolution Evaluation goes a step further, from diagnostic to behavioral: it presents agents with controlled snapshot sequences to simulate the self-evolution process, and evaluates their performance on subsequent episodes. BenchTrace also addresses the controllability issue, along two dimensions. From the perspective of evolution, we can construct specific snapshot sequences to isolate particular patterns of evolution. From the perspective of cross-model comparison, any agent framework can be evaluated on the same sequence of evolution snapshots, making the evaluation model-agnostic and enabling fair comparison across methods. In summary, this paper makes the following contributions. 1 1 1 Our code is available at https://github.com/Alab-NII/BenchTrace and the dataset at https://huggingface.co/datasets/huangjh16/BenchTrace . (i) We introduce a high-quality snapshot-reflection dataset spanning diverse tasks, where each snapshot is annotated with the core failure instances responsible for task failure. (ii) BenchTrace provides a Reflection Evaluation that assesses the base LLM’s reflection ability through QA tasks, and an Evolution Evaluation that assesses the agent’s evolution effectiveness through controlled snapshot sequences, together addressing the interpretability and controllability issues. (iii) Applying BenchTrace to current self-evolution methods reveals two concrete limitations in current self-evolution agents: (a) agents forget early lessons as noise episodes accumulate, and (b) agents fail to generalize their reflections beyond the specific context, causing negative transfer across task contexts. 2 Related Work 2.1 Self-Evolving Agents ReAct ( 19 ) augments LLMs with interleaved reasoning and action steps but accumulates no experience across episodes. Self-evolving agents address this by enabling cross-episode learning, keeping the model frozen and evolving the agent’s context or memory. Instance-based methods store past episodes and retrieve relevant ones at query time: RAG ( 10 ) retrieves by semantic similarity, while ReMem ( 16 ) and MemRL ( 21 ) improve upon this through meta-reasoning over stored memories and value-based retrieval policies, respectively. Abstraction-based methods distill past failures into verbal lessons: Reflexion ( 12 ) generates self-critiques as episodic memory, while EvoTest ( 8 ) combines both strategies by jointly updating memory and rewriting the guiding prompt. AutoSkill ( 18 ) takes a different approach, accumulating versioned behavioral skill artifacts across sessions. 2.2 Benchmarks for Evaluating Self-Evolving Agents Existing benchmarks offer two types of tasks. Environment-based tasks place agents in simulated worlds requiring sequential action execution, spanning text-based games ( 7 ) , household manipulation ( 13 ) , grid navigation ( 3 ) , and science experiments ( 15 ) . Information-based tasks require handling information-intensive queries across sessions: Evo-Memory ( 16 ) covers QA and tool-use streams, while MemoryArena ( 9 ) tests cross-session retention through web shopping, travel planning, and formal reasoning. Both types, however, rely on aggregate outcome metrics and provide no ground-truth annotations of what agents should have learned from past failures, making it impossible to assess reflection quality directly. 3 The BenchTrace Benchmark 3.1 Overview BenchTrace consists of two components. The Dataset is a collection of snapshots, each paired with ground-truth reflection annotations along the detection, localization, and diagnosis hierarchy. The Evaluation Suite is built on top of the dataset and comprises two types of evaluation:

Read the complete text