SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment

Sihang Jiang, Lipeng Ma, Zhonghua Hong, Keyi Wang, Zhiyu Lu, Shisong Chen · 2026

Read SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment by Sihang Jiang, Lipeng Ma, Zhonghua Hong, Keyi Wang, Zhiyu Lu, Shisong Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08988.txt.

Introduction

Jiang and eleven colleagues formalize the self-evolving agent, which should carry experience across tasks instead of starting fresh each time. Current agents handle single episodes well but forget lessons across tasks, trapped by fixed toolsets and episodic amnesia. The paper defines this agent type and proposes the Evolutionary Flywheel as its minimal architecture. SEA-Eval is its first benchmark, built to judge such agents. Tests show token use differs by up to 31.2 times between frameworks at similar success rates, so success alone misleads.

Catalog record

CREATORSihang Jiang, Lipeng Ma, Zhonghua Hong, Keyi Wang, Zhiyu Lu, Shisong Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70159
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08988

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment Sihang Jiang, Lipeng Ma, Zhonghua Hong, Keyi Wang, Zhiyu Lu, Shisong Chen Abstract Current LLM-based agents demonstrate strong performance in episodic task execution but remain constrained by static toolsets and episodic amnesia, failing to accumulate experience across task boundaries. This paper presents the first formal definition of the Self-Evolving Agent (SEA), formalizes the Evolutionary Flywheel as its minimal sufficient architecture, and introduces SEA-Eval—the first benchmark designed specifically for evaluating SEAs. Grounded in Flywheel theory, SEA-Eval establishes S ​ R SR and T T as primary metrics and enables through sequential task stream design the independent quantification of evolutionary gain, evolutionary stability, and implicit alignment convergence. Empirical evaluation reveals that under identical success rates, token consumption differs by up to 31.2 × \times across frameworks, with divergent evolutionary trajectories under sequential analysis—demonstrating that success rate alone creates a capability illusion and that the sequential convergence of T T is the key criterion for distinguishing genuine evolution from pseudo-evolution. 1 Introduction Large Language Model (LLM) agents have emerged as a pivotal paradigm for translating generative intelligence into tangible productivity. Since the inception of tool-augmented reasoning, agents have demonstrated substantial industrial utility across domains such as autonomous software engineering, scientific discovery, and complex workflow orchestration. Recent advancements in system-native architectures, exemplified by frameworks like GenericAgent 1 1 1 https://github.com/lsdefine/GenericAgent and OpenClaw 2 2 2 https://github.com/openclaw/openclaw , have propelled agents beyond the confines of text-based reasoning engines. By achieving deep integration with operating systems and native applications, these agents are transitioning toward a state of digital embodiment : no longer mere semantic processors, but functional actors capable of sustaining high-load industrial tasks directly upon the underlying digital infrastructure. This evolutionary trajectory points toward a new agentic paradigm: the Self-Evolving Agent (SEA) . The core characteristics of an SEA are defined by two properties. First, complex interaction in open-ended environments : the agent operates directly upon raw GUI pixels, dynamic DOM trees, and low-level system interfaces, unconstrained by predefined API boundaries. Second, continuous, multi-turn evolution : by maintaining persistent memory across task boundaries, the agent transforms historical interaction experience into cognitive priors that reconfigure the execution of subsequent tasks. Architecturally, the SEA is centered on the Evolutionary Flywheel —a closed-loop mechanism formed by the symbiotic coupling of an Execution Hub and a Cognition Hub, operating through three sequential phases of execution, distillation, and augmented execution. When the flywheel operates effectively, execution overhead converges monotonically as task frequency increases; when it fails, the agent falls into pseudo-evolution , relying on zero-shot reasoning with execution overhead scaling linearly with task frequency. However, the novel characteristics of SEAs expose a fundamental mismatch with existing evaluation frameworks. Current benchmarks adopt the isolated task episode as the basic unit of evaluation, and are built around the episodic paradigm in which memory is reset at every task boundary—making the assessment of cross-task knowledge accumulation and continuous capability growth structurally impossible. This mismatch manifests across two dimensions: sequential task stream designs are absent, making it structurally impossible to capture cross-task evolutionary dynamics; and success-rate-dominated static metrics cannot capture the dynamic convergence of execution efficiency as experience accumulates, rendering pseudo-evolution entirely invisible to existing evaluation frameworks ( 30 ; 2 ; 5 ) . This produces a capability illusion in practice: two frameworks achieving comparable success rates may differ fundamentally in evolutionary quality, yet existing benchmarks cannot distinguish them. To bridge this evaluative gap, we introduce SEA-Eval ---the first evaluation benchmark 3 3 3 https://github.com/LeaperOvO/SEA-Eval designed for self-evolving agents. SEA-Eval evaluates agents across two complementary dimensions: intra-task execution reliability , assessing immediate execution capability under environmental uncertainty independent of prior experience; and long-term evolutionary performance , assessing evolutionary gain, evolutionary stability, and implicit alignment convergence across sequential task deployments. In terms of metrics, we establish SR and T as the primary evaluative primitives, a selection grounded directly in the Evolutionary Flywheel framework: SR serves as the minimum reliability guarantee for task completion, while the sequential convergence trajectory of T serves as the core signal for distinguishing genuine evolution from pseudo-evolution. The dataset comprises 32 atomic tasks organized by three difficulty tiers and systematically composed into three types of sequential streams—correlated, orthogonal, and implicit intent—under two noise conditions that decouple retrieval robustness from evolutionary capability as an independent control variable. Based on this framework, we conduct an empirical study evaluating leading agent frameworks including OpenClaw and GenericAgent . The results reveal a striking Evolutionary Bottleneck . In static evaluation, both frameworks achieve a 100% task success rate, yet their token consumption differs by approximately sevenfold in aggregate—OpenClaw consumes a total of 27,364K tokens across all tasks, while GenericAgent consumes only 3,918K tokens, with the gap reaching up to 31.2 \times on individual tasks. In sequential task analysis, GenericAgent exhibits genuine evolutionary characteristics: in the academic database retrieval task, its token consumption converges monotonically from 520K to 117K, reflecting the effective transformation of experience into low-cost execution strategies. OpenClaw, by contrast, displays the hallmarks of pseudo-evolution: token consumption fluctuates across sequential task variants, fails to form a stable convergence trajectory, and collapses entirely on certain sequential tasks. This contrast demonstrates that success rate alone creates an illusion of capability, while the sequential convergence of token consumption is necessary to faithfully reveal an agent’s evolutionary quality. The contributions of this work are three-fold: • We are the first to present a formal definition of Self-Evolving Agents grounded in digital embodiment and continuous cross-task evolution. We formalize the Evolutionary Flywheel as the minimal sufficient architecture of the SEA, which for the first time provides a theoretically grounded criterion for distinguishing genuine evolution from pseudo-evolution, and characterizes two failure modes—distillation failure and retrieval failure. • We introduce SEA-Eval, the first benchmark designed specifically for evaluating SEAs. We are the first to establish SR and T as a Flywheel-grounded metric system, which jointly covers two evaluation dimensions (intra-task execution reliability and long-term evolutionary performance). The accompanying dataset comprises 32 atomic tasks organized across three difficulty tiers and systematically composed into three types of sequential evaluation units, providing sufficient coverage and task diversity to support precise and comprehensive evaluation. • We reveal a significant evolutionary bottleneck among mainstream frameworks: under identical success rates in static evaluation, token consumption differs by up to 31.2 \times on individual tasks, with divergent evolutionary trajectories emerging under sequential analysis, which further demonstrates the effectiveness of SEA-Eval. 2 Related Work 2.1 LLM Agents and Episodic Amnesia The development of autonomous agents has progressed from restricted, rule-based systems to increasingly open-ended and adaptive entities ( 22 ; 5 ; 24 ) . The emergence of LLMs has catalyzed a new generation of agents capable of interleaving reasoning, planning, and tool use, forming iterative reasoning–action loops that enable interaction with complex environments ( 28 ; 17 ) . However, current LLM-based agents remain fundamentally constrained by episodic amnesia. We extend this concept from cognitive science to the agent lifecycle: just as patients with episodic amnesia retain procedural skills yet cannot encode specific experiences into persistent memory, LLM agents demonstrate strong within-task reasoning yet reset their cognitive state at every task boundary, failing to transform interaction experience into reusable cross-task cognitive assets ( 12 ; 4 ) .

Read the complete text