AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems

Boxuan Zhang, Jianing Zhu, Zeru Shi, Dongfang Liu, Ruixiang Tang, [2pt] Rutgers University · 2026

Read AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems by Boxuan Zhang, Jianing Zhu, Zeru Shi, Dongfang Liu, Ruixiang Tang, [2pt] Rutgers University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08715.txt.

Introduction

Boxuan Zhang and five coauthors catch failing agent runs while they can still be saved, instead of assigning blame after the fact. In multi agent systems, one decisive error often passes downstream and ruins the whole trajectory. AgentForesight reframes detection as online auditing: an auditor sees only the steps so far and must alarm at the earliest decisive error or let the run continue.

A curated corpus of two thousand trajectories across coding, math, and agent tasks marks decisive errors by multi judge consensus. Their compact auditor trains in two stages to anticipate risk, then localize the faulty step, agent, and nature. It beats leading commercial models with far lower localization error.

Catalog record

CREATORBoxuan Zhang, Jianing Zhu, Zeru Shi, Dongfang Liu, Ruixiang Tang, [2pt] Rutgers University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08715

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems Boxuan Zhang, Jianing Zhu, Zeru Shi, Dongfang Liu, Ruixiang Tang, [2pt] Rutgers University Abstract LLM-based multi-agent systems are increasingly deployed on long-horizon tasks, but a single decisive error is often accepted by downstream agents and cascades into trajectory-level failure. Existing work frames this as post-hoc failure attribution , diagnosing the responsible agent and step after the trajectory has ended. However, this paradigm forfeits any opportunity to intervene while trajectory is still unfolding. In this work, we introduce AgentForesight , a framework that reframes this problem as online auditing : at each step of an unfolding trajectory, an auditor observes only the current prefix and must either continue the run or alarm at the earliest decisive error without access to future steps. To this end, we curate AFTraj- 2 2 K , a corpus of agentic trajectories across Coding, Math, and Agentic domains, in which safe trajectories are retained under a strict curation pipeline and unsafe trajectories are annotated at the step of their decisive error via consensus among multiple LLM judges. Built on that, we develop AgentForesight -7B, a compact online auditor trained with a coarse-to-fine reinforcement learning recipe that first equips it with a risk-anticipation prior at the failure boundary on adjacent safe/unsafe prefix pairs, then sharpens this prior into precise step-level localization under a three-axis reward jointly targeting the what , where , and who of an audit verdict. Across AFTraj- 2 2 K and an external Who&When benchmark, AgentForesight -7B outperforms leading proprietary models, including GPT-4.1 and DeepSeek-V4-Pro, achieving up to +19.9% performance gain and 3 × \times lower step localization error, opening the loop from post-hoc failure detection to enabling deployment-time intervention. Project Page: https://zbox1005.github.io/agent-foresight/ 1 Introduction Large language models (LLMs) have rapidly evolved into agentic systems that plan, reason, and act across long-horizon tasks through coordinated tool use and inter-agent communication [ 59 , 53 , 17 , 28 ] . By decomposing complex objectives into specialized sub-tasks, these systems now tackle problems once considered out of reach, spanning software development [ 20 , 51 ] , scientific discovery [ 11 , 12 ] , and open-ended web navigation [ 66 , 33 ] . However, such gains in capability come with a structural cost. Since each step is conditioned on earlier outputs, a single decisive error , e.g., a malformed tool call or a flawed intermediate deduction, is easily accepted by downstream agents and cascades into a full-trajectory failure [ 3 , 63 , 25 ] . Once deployed in real-world environments with access to APIs and external services, such failures extend beyond benchmark accuracy into unanticipated operational risks [ 60 , 45 ] , making reliability a central bottleneck for the deployment of LLM multi-agent systems. Although prior work has recognized failure analysis as a central concern for reliable LLM multi-agent systems, existing approaches predominantly frame it as post-hoc failure attribution , asking which agent or step is responsible once the trajectory has already failed [ 63 , 62 , 67 ] , as illustrated in Figure 1 (a). For instance, Who&When [ 63 ] and AgenTracer [ 62 ] curate failed trajectories and train or prompt models to pinpoint the decisive error step after the run has ended, while AgentDebug [ 67 ] and related debugging frameworks [ 49 , 19 ] analyze full trajectories to taxonomize failures and supply corrective feedback for subsequent retries. However, confining failure analysis to the post-hoc regime forgoes any opportunity to act while the trajectory is still unfolding. Before a diagnosis is available, agents have already consumed further tool calls and external resources, and in deployment settings may have triggered irreversible side effects. This naturally motivates a fundamental research question: Can we audit unfolding prefixes rather than completed trajectories to catch decisive errors before propagation locks in failure? To answer this question, we introduce online auditing , where a dedicated auditor commits a continue-or-alarm verdict at every step of an unfolding trajectory, as illustrated in Figure 1 (b). Concretely, instead of inspecting a completed trajectory with full hindsight, the auditor sees only the current prefix at each step and must judge it without access to future steps, tool responses, or the eventual outcome. This reframe turns failure analysis from a passive post-hoc diagnosis of completed runs into an active safeguard that can intervene before downstream propagation locks in the failure. Operationalizing it places two new demands on the auditor: ① it must reliably separate prefixes that are still safe from those already past a decisive error , and ② it must commit at the very step the error occurs, not in hindsight. Both demands exceed what existing failure-attribution data or models can provide, motivating the creation of both a new dataset and a dedicated training recipe. Figure 1: Comparison of (a) post-hoc failure attribution and (b) online auditing on the same multi-agent task. (a) Post-hoc failure attribution inspects the trajectory only after it has failed and identifies the decisive error retrospectively, by which point downstream propagation has already locked in the failure. (b) Our AgentForesight instead evaluates each prefix as the trajectory unfolds and flags the decisive error at the very step it commits, opening an intervention window before the failure is locked in (see Section 2 ). To instantiate this formulation, we develop AgentForesight , a framework that addresses these two demands through a dedicated dataset and a coarse-to-fine training recipe. We first construct AFTraj- 2 K , a curated corpus of agentic trajectories spanning Coding, Math, and Agentic domains, pairing safe trajectories retained under a strict filtering pipeline with failure trajectories annotated at their decisive error step under multi-judge voting verification. Building on the curated dataset, we fine-tune Qwen2.5-7B-Instruct via reinforcement learning to obtain AgentForesight -7B, a compact online auditor first equipped with a risk-anticipation prior at the failure boundary on adjacent safe/unsafe prefix pairs, then sharpened into precise step-level localization under a three-axis reward jointly targeting the structure of verdict ( what ), the timing of alarm ( where ), and the responsible agent ( who ). Together, AgentForesight -7B runs alongside off-the-shelf multi-agent systems and issues step-level continue-or-alarm verdicts on unfolding trajectories, without retraining the underlying agentic system. We extensively evaluate AgentForesight -7B on AFTraj- 2 K and the external Who&When [ 63 ] benchmark, where it surpasses both its Qwen2.5-7B-Instruct base model and leading proprietary judges including GPT-4.1 and DeepSeek-V4-Pro, achieving +19.9\% higher Exact-F1 and 3\times lower step localization error than the strongest proprietary baseline. These gains confirm that our coarse-to-fine recipe yields a compact online auditor that outperforms much larger proprietary judges under the prefix-restricted online setting. We summarize our contributions as follows: • We introduce online auditing , a deployment-time reframing of agentic failure analysis that audits unfolding trajectories step by step rather than diagnosing them after failure (Section 2 ). • We construct AFTraj- 2 K , a curated corpus of agentic trajectories spanning Coding, Math, and Agentic domains, pairing strictly filtered safe runs with multi-judge verified failure runs annotated at their decisive error step (Section 3.1 ). • We develop AgentForesight -7B, a compact online auditor trained via a coarse-to-fine RL recipe that first equips it with

Read the complete text