Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation

Hongliu CAO (✉), Ilias DRIOUICH, Eoin THOMAS · 2026

Read Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation by Hongliu CAO (✉), Ilias DRIOUICH, Eoin THOMAS (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03116.txt.

Introduction

Hongliu Cao, Ilias Driouich, and Eoin Thomas show that an AI agent can finish the task and still fail the job. Benchmarks usually score only completion, yet agents in high-stakes settings also cut corners, break policy, or misread intent along the way. Their 2026 framework, Procedure-Aware Evaluation, records the agent's procedure as structured observations.

Scoring runs along four axes: utility, efficiency, interaction quality, and procedural integrity, with hard gates that disqualify corrupt outcomes outright. On a standard agent benchmark, between about a quarter and three quarters of reported successes hid violations. Gating those out collapsed headline scores and reshuffled model rankings. Each model also showed a distinct failure signature, and the benchmark itself carried design flaws the analysis exposed.

Catalog record

CREATORHongliu CAO (✉), Ilias DRIOUICH, Eoin THOMAS
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89542
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03116

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation Hongliu CAO (✉), Ilias DRIOUICH, Eoin THOMAS Abstract Large Language Model (LLM)-based agents are increasingly adopted in high-stakes settings, but current benchmarks evaluate mainly whether a task was completed, not how . We introduce Procedure-Aware Evaluation (PAE), a framework that formalizes agent procedures as structured observations and exposes consistency relationships between what agents observe, communicate, and execute. PAE evaluates agents along complementary axes (Utility, Efficiency, Interaction Quality, Procedural Integrity) and applies multi-dimensional gating that categorically disqualifies corrupt outcomes. Evaluating state-of-the-art LLM agents on τ \tau -bench yields findings at the axis, compliance, and benchmark levels. At the axis level, the dimensions capture non-redundant failure modes: utility masks reliability gaps, speed does not imply precision, and conciseness does not predict intent adherence. At the procedural compliance level, 27–78% of benchmark reported successes are corrupt successes concealing violations across interaction and integrity. Furthermore, gating substantially collapses Passˆ4 rate and affects model rankings. The analysis of corrupt success cases reveals distinctive per-model failure signatures: GPT-5 spreads errors across policy, execution, and intent dimensions; Kimi-K2-Thinking concentrates 78% of violations in policy faithfulness and compliance; and Mistral-Large-3 is dominated by faithfulness failures. At the benchmark level, our analysis exposes structural flaws in the benchmark design, including task scope gaps, contradictory reward signals, and simulator artifacts that produce accidental successes. 1 Introduction LLM-based agents have evolved from single-turn text generators into autonomous systems that coordinate multi-step reasoning, maintain shared state, and invoke external tools for complex real-world tasks [ 37 ] . This transition from text generation to task execution marks a fundamental shift in AI system design. It also exposes a critical gap in evaluation methodology [ 42 , 35 , 32 ] . Current benchmarks assess agents primarily through outcome metrics (binary success rates, terminal-state matching, or tool-call accuracy) [ 22 ] , while ignoring the procedures by which outcomes are achieved. This creates a dangerous blind spot: an agent that completes a task by bypassing authorization, fabricating confirmations, or communicating incorrect policy is scored identically to one that follows every required step. Existing evaluation frameworks cannot distinguish these cases because they conflate what is achieved with how it is achieved. As agents move into production deployments involving payments, personal data, and policy enforcement, this blind spot becomes a deployment risk. We address it with Procedure-Aware Evaluation (PAE), a framework that formalizes agent procedures as structured observations and evaluates them along four axes (§ 3 ): what was achieved (Utility), at what cost (Efficiency), whether the user was correctly informed and well served (Interaction Quality), and whether agent actions complied with policies and agent statements were grounded in observed data (Procedural Integrity). Six-dimension gating across interaction and integrity metrics exposes corrupt successes , i.e., task completions achieved through violations that production systems must categorically reject. Applying PAE to \tau -bench yields three key findings: 1. Multi-axis evaluation. Evaluating GPT-5, Kimi-K2-Thinking, and Mistral shows the four axes capture complementary, non-redundant failure modes: success rates do not reflect true reliability, speed does not imply precision, conciseness does not predict intent adherence, and integrity is orthogonal to all three (§ 4 ). 2. Corrupt success exposure. Gating reveals 27–78% of successes are procedurally corrupt, collapsing gated Passˆ4 to 2–24%, reversing model rankings, and leaving no model above 24% procedurally compliant reliability. Manual analysis of 131 cases confirms distinctive per-model failure signatures: GPT-5 spreads errors broadly, Kimi concentrates 78% in policy faithfulness and compliance, and Mistral is dominated by faithfulness failures (data and policy faithfulness combined) (§ 5 ). 3. Benchmark scrutiny. The same manual analysis exposes structural flaws in the benchmark itself: task definitions that omit legitimate agent paths, contradictory reward signals, and simulator artifacts that produce accidental successes (§ 5.2 ). 2 Related Work 2.1 Evaluation of LLM Agents There is a growing body of work on the evaluation of LLM based agents. Recent surveys [ 22 , 37 ] propose multiple dimensions for organizing this space, reflecting the increasing complexity of agent behavior and deployment contexts. Building on these frameworks, we structure the literature around three evaluation objectives: agent behavior, agent capability, and reliability (robustness, safety, fairness) inspired by a recent survey [ 22 ] . Agent behavior evaluation examines how well an agent meets end - user expectations through metrics such as task completion and output quality, both of which emphasize outcome oriented aspects [ 22 ] . The dominant metrics assess task completion with Success Rate or Overall Success Rate [ 7 , 5 , 39 ] . Benchmarks such as SWE-bench [ 15 ] (resolving GitHub issues), WebArena [ 42 ] (web navigation), GAIA [ 21 ] (general assistance), and \tau -bench [ 36 ] (customer service) all prioritize whether the agent reached the correct end state. Output quality metrics (fluency, coherence, factual correctness [ 28 , 40 , 10 ] ) extend this to response characteristics. Procedure-relevant metrics in this category remain sparse. Latency and cost metrics (Time-to-First-Token, token usage) implicitly measure interaction efficiency (a procedural concern) but are rarely integrated into benchmark scoring [ 22 ] . More critically, how an agent achieves success (e.g., turns required, clarification burden) is typically unreported. Capability evaluation targets specific capabilities of LLM based agents, including tool use, planning and reasoning, memory and context retention, and multi agent collaboration [ 37 , 30 , 13 ] . Tool use evaluation (e.g. ToolBench [ 25 ] , APIBench [ 24 ] ) checks whether agents invoke correct functions with valid parameters. Planning metrics (Node F1, Edge F1, Progress Rate) assess whether agents select and sequence tools appropriately [ 6 , 5 , 30 ] . These metrics are inherently procedure-focused : they audit the trajectory, not just the terminal state. However, these trajectory checks assess functional correctness (did the agent execute the right tool calls with right parameters?) rather than procedural conformance (did the agent communicate right information and respect constraints: authorization, data minimization, escalation protocols?). Reliability metrics assess consistency across repetitions [ 19 ] with metrics such as pass@ k and passˆ k in [ 36 ] , and robustness to input perturbations (HELM [ 18 ] ) as well as to failures handling [ 29 ] . Safety benchmarks (Agent-safetybench [ 41 ] , AgentHarm [ 1 ] , ST-WebAgentBench [ 17 ] , CoSafe [ 38 ] ) evaluate toxicity, bias, adversarial resistance etc. Several works also highlight fairness concerns in agentic systems [ 4 , 3 , 26 ] . These remain largely outcome-focused : reliability tests whether outputs remain stable; safety tests whether agents refuse harmful instructions or produce toxic content. Procedural gaps span several dimensions. Existing evaluations rarely test whether an agent’s task-solving procedure is stable, recoverable, or robust; they overlook whether different users experience equitable clarification load or consistent rule enforcement; and they focus on final safety outcomes while ignoring unsafe intermediate actions and privacy risks. 2.2 The Imperative for Procedure Auditing Despite progress toward interactive and safety-aware benchmarks, existing metrics focus on task success while ignoring whether the agent followed proper procedure. Three structural risks motivate procedure auditing. First, corrupt success : an agent may reach the correct terminal state while violating mandatory constraints. Second,

Read the complete text