CORRECT: Condensed Error Recognition via Knowledge Transfer in Multi-agent Systems

Yifan Yu, Moyan Li, Shaoyuan Xu, Jinmiao Fu, Xinhai Hou, Fan Lai, Bryan Wang · 2025

Read CORRECT: Condensed Error Recognition via Knowledge Transfer in Multi-agent Systems by Yifan Yu, Moyan Li, Shaoyuan Xu, Jinmiao Fu, Xinhai Hou, Fan Lai, Bryan Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24088.txt.

Introduction

When many AI agents work on one task, a small slip by one agent can spread and wreck the whole job. Yifan Yu and 6 co-authors present CORRECT, a light tool that spots where such errors occur. Their key point is that failures, though logs differ, often share the same structural patterns.

CORRECT keeps a cache of distilled error patterns and reuses them on new runs, so no retraining is needed. The team also built a data set of over two thousand labeled failure runs, made by injecting errors and checked by humans. Across seven systems, CORRECT improved step-level error spotting by up to about a fifth, with almost no added cost.

Catalog record

CREATORYifan Yu, Moyan Li, Shaoyuan Xu, Jinmiao Fu, Xinhai Hou, Fan Lai, Bryan Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24088

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CORRECT: Condensed Error Recognition via Knowledge Transfer in Multi-agent Systems Yifan Yu, Moyan Li, Shaoyuan Xu, Jinmiao Fu, Xinhai Hou, Fan Lai, Bryan Wang Abstract Multi-agent systems (MAS) are increasingly capable of tackling complex real-world tasks, yet their reliance on inter-agent coordination, tool use, and long-horizon reasoning makes error recognition particularly challenging. Minor errors can propagate across agents, escalating into task failures while producing long, intertwined execution trajectories that impose significant costs for both human developers and automated systems to debug and analyze. Our key insight is that, despite surface differences in failure trajectories (e.g., logs), MAS errors often recur with similar structural patterns. This paper presents CORRECT , the first lightweight, training-free framework that leverages an online cache of distilled error schemata to recognize and transfer knowledge of failure structures across new requests. This cache-based reuse allows LLMs to perform targeted error localization at inference time, avoiding the need for expensive retraining while adapting to dynamic MAS deployments in subseconds. To support rigorous study in this domain, we also introduce CORRECT -Error, a large-scale dataset of over 2,000 annotated trajectories collected through a novel error-injection pipeline guided by real-world distributions, and further validated through human evaluation to ensure alignment with natural failure patterns. Experiments across seven diverse MAS applications show that CORRECT improves step-level error localization up to 19.8% over existing advances while at near-zero overhead, substantially narrowing the gap between automated and human-level error recognition. 1 Introduction Multi-agent systems (MAS) have emerged as a powerful paradigm for solving complex tasks that require diverse capabilities and collaborative problem-solving, with demonstrated success in various domains, including software development ( Qian et al. 2023 ; Hong et al. 2023 ; Zhang et al. 2024 ) , scientific research ( Lu et al. 2024 ) , web navigation ( Zhou et al. 2023 ) , and general-purpose task automation ( Wu et al. 2024 ; Wang et al. 2025 ) . By orchestrating multiple specialized agents, MAS can tackle challenges beyond the reach of single-agent systems (SAS) that rely on single LLMs for task solving, achieving human-level performance on benchmarks like GAIA ( Mialon et al. 2023 ) and SWE-bench ( Jimenez et al. 2023 ) . However, as MAS increasingly scale in both complexity (e.g., sophisticated interactions with other agents and tools) and deployments, decisive error recognition in MAS deployments becomes exceptionally challenging. Unlike SAS where failures can be traced to a single faulty output, MAS failures often emerge from cascading effects across agents: a single error-prone step by one agent can propagate through downstream interactions, ultimately causing task failure ( Gao et al. 2025 ) . We refer to this fundamental challenge as decisive error recognition : pinpointing the precise agent and step that first triggered the failure ( Zhang et al. 2025 ) . Efficient decisive error recognition is essential for reliable MAS deployments, sustaining service quality, and guiding operational management such as safe agent upgrades, targeted restarts, and service monitoring ( Epperson et al. 2025 ) . Unfortunately, decisive error recognition in MAS remains open-ended due to three fundamental obstacles: (1) Generality : MAS span a wide spectrum of applications, and even within an application, requests can exhibit drastically different error patterns, making it difficult to design methods that generalize. Existing advances ( Zhang et al. 2025 ) often resort to LLM-as-a-judge methods for generality, yet achieve \leq 10% accuracy in pinpointing the exact error step, barely above the accuracy of random guessing (3%). Recent efforts ( Ge et al. 2025 ) to improve accuracy through fine-tuning not only hurt generalization, but fall short in (2) Data Efficiency : obtaining labeled data in error recognition is notoriously expensive and inherently ambiguous. For example, in the Who&When dataset ( Zhang et al. 2025 ) , annotators spent over 30 expert hours labeling fewer than 200 trajectories, yet disagreement rates exceeded 50%. This makes any training-based approaches ineffective without voluminous data; and (3) Computation Efficiency : even with sufficient data, tuning LLMs for error recognition may be impractical to catch up with deployments where new error types arise continuously and sporadically, e.g., new attacks in cloud AIOps ( Wang et al. 2025 ) . In this paper, we first notice that failures in MAS tend to recur with similar structures across requests (§ 2.2 ). Because a MAS application often relies on the same role specifications, orchestration rules, tool APIs, and verification policies, diverse requests often funnel into common decision skeletons. Our real-world analysis of Who&When dataset proposed in Zhang et al. 2025 shows that over 80% of failure trajectories have at least one counterpart with \geq 0.8 semantic similarity in error logs. This suggests that error knowledge can be systematically distilled, cached, and reused . However, naive approaches, such as in-context learning (ICL) that insert prior trajectories as in-context exemplars, quickly break down: logs can exceed 32,000 tokens ( Yang et al. 2025 ; Dubey et al. 2024 ) , contain low-entropy noise, and even underperform zero-shot baselines (§ 2.2 ). These challenges demand a new approach: one that is general, data-efficient, and lightweight for practical deployments. In this paper, we propose CORRECT ( CO ndensed e R ror REC ognition via knowledge T ransfer), a novel framework that formalizes decisive error recognition as a first-class problem for reliable MAS, and automatically distills prior error patterns into compact, reusable schemas that capture their essential signatures, triggering contexts, and propagation patterns. During inference, CORRECT identifies relevant schemas and applies them to the new request, enabling accurate, lightweight, and training-free recognition in dynamic environments. Our contributions are summarized as follows: • CORRECT: the first schema-guided detector with broad generality. We introduce CORRECT , the first framework that distills recurrent MAS failures into compact error schemata and reuses them for decisive error recognition. Unlike LLM-as-a-judge approaches that trade accuracy for generality, or costly fine-tuning approaches, CORRECT leverages schemata to transfer knowledge across requests in real time. This design improves step-level localization accuracy up to 20 points over state-of-the-art designs ( Zhang et al. 2025 ) , while remaining training-free and computationally efficient, making it practical for deployment in dynamic MAS environments. • CORRECT -Error: a versatile, large-scale, and high-fidelity dataset for MAS error recognition. To close the benchmarking gap, we construct CORRECT -Error, a large-scale collection of over 2,000 multi-agent trajectories with fine-grained, step-level error annotations. Unlike prior datasets that are small and noisy, CORRECT -Error is built using a novel error-injection pipeline guided by real-world failure distributions, yielding trajectories that balance controlled coverage with the realism of natural MAS failures. We conduct extensive human validation, confirming strong alignment between synthetic annotations and expert judgment. CORRECT -Error not only enables rigorous evaluation of CORRECT , but also establishes a reusable, extensible benchmark for the community to advance the development of MAS error recognition methods. Together, these contributions advance the state of the art in MAS error recognition and establish a new foundation and datasets for making MAS more reliable, interpretable, and deployable at scale. 2 Background and Motivation 2.1 Decisive Error in Multi-Agent Systems Task failures in MAS often arise from specific decisive errors that, once committed, render successful task completion impossible. We formalize this notion following prior advances ( Zhang et

Read the complete text