ContractBench : Can LLM Agents Preserve Observation Contracts?
Jicheng Wang, Yifeng He, Zili Wang, [1pt] Hanwen XingArkaprava De, Hao Chen, [2pt] University of California, Davis, [2pt] Corresponding author. Equal contribution. · 2026
Read ContractBench : Can LLM Agents Preserve Observation Contracts? by Jicheng Wang, Yifeng He, Zili Wang, [1pt] Hanwen XingArkaprava De, Hao Chen, [2pt] University of California, Davis, [2pt] Corresponding author. Equal contribution. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17281.txt.
Introduction
Wang and colleagues show that tool-using language-model agents often mishandle observation contracts, artifacts like signed links and session tokens whose later use is time-bound and byte-sensitive. Their ContractBench benchmark probes expiry and corruption failures across 33 tasks and 38 models. No model clears 80 percent, scaling is uneven across families, and their failure taxonomy as an in-context signal lifts paired failures by 7.1 points.
Catalog record
| CREATOR | Jicheng Wang, Yifeng He, Zili Wang, [1pt] Hanwen XingArkaprava De, Hao Chen, [2pt] University of California, Davis, [2pt] Corresponding author. Equal contribution. |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70049 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17281
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ContractBench : Can LLM Agents Preserve Observation Contracts? Jicheng Wang, Yifeng He, Zili Wang, [1pt] Hanwen XingArkaprava De, Hao Chen, [2pt] University of California, Davis, [2pt] Corresponding author. Equal contribution. Abstract Tool-augmented LLM agents call APIs whose intermediate outputs, such as presigned URLs, session tokens, and OAuth state parameters, are observation contracts : artifacts whose later use is constrained by the external system that produced them. We show that observation contract compliance (preserving the temporal validity and byte-level integrity) is an emergent, regression-prone capability: it is neither guaranteed by general tool-use ability nor consistently improved by larger or newer models. To measure this, we introduce ContractBench , a benchmark of 33 dual-axis tasks that probe two orthogonal failure modes no existing benchmark evaluates: validity failures (using an artifact after expiry) and integrity failures (corrupting an artifact’s bytes through the observation-to-action pipeline). Our evaluation is deterministic and programmatic, with a virtual clock controls time and SHA-256 hashes verify byte integrity. We assign the outcomes failure labels drawn from real-world API specifications. We evaluate 38 models and report four findings: (i) no evaluated model clears 80%, with Claude-Opus-4.6 leading at 77.8%, revealing that current frontier models still fail to comply with observation contracts. (ii) a sharp within-family capability cliff in Qwen 3.5 between 4B (0%) and 9B (56.6%), smoothing to 70.7% at 397B-A17B: what emerges across the cliff is mid-trajectory restraint, not tool-call competence; (iii) non-monotonic scaling across GPT-5 family: agentic post-training can erode compliance through sycophancy-driven regression; (iv) and our failure taxonomy works as an actionable in-context reward signal, yielding + 7.1 +7.1 pp on 42 paired GPT-5.1 failures. 1 Introduction Large language model (LLM) agents are increasingly deployed as autonomous assistants by personal users and enterprises ( 49 ; 56 ) . To reach a user-specified goal, an LLM agent calls external networked APIs, navigates websites, executes shell commands, and reads or writes files, observing the outcome of each step before choosing the next. These agents are also growing more autonomous and longer-lived, with recent systems completing everyday online tasks across dozens of live platforms with minimal human oversight ( 57 ) . In production, they already resolve software-engineering issues ( 54 ) , handle multi-turn customer-service dialogues ( 55 ) , and carry out complex terminal tasks in scientific computing and cybersecurity ( 30 ) . In this setting, a wrong action can break a workflow, corrupt a database, or cause an irreversible side effect ( 21 ) . Executing reliably matters as much as understanding the task. Several benchmarks now ground agent evaluation in real APIs, codebases, and user interactions: web navigation ( 59 ) , software engineering ( 22 ; 30 ) , general tool use ( 46 ; 27 ) , user interaction under domain-specific policies ( 55 ) , and multi-constraint planning ( 53 ) . In these benchmarks, the intermediate steps of a multi-step task do not impose constraints on the steps that follow: a tool output is information for the agent’s next decision, and the agent can paraphrase, summarize, or even lose it without penalty so long as the workflow reaches the right end state. Real API workflows behave differently. The output of one step often dictates exactly how the agent must execute the next step. For example, the agent must relay a signed URL byte-for-byte, use a token before it expires, and round-trip an OAuth state parameter unmodified. Current benchmarks do not test whether the agent respects these inter-step constraints. Yet this is a primary failure mode in deployment: a single intermediate-step violation invalidates the whole task flow. Previous AI agent evaluations have overlooked this setting, but it is routine in enterprise API workflows: a tool returns an artifact, and the external system that produced it constrains its later use. We call such a tool-returned artifact an observation contract . Consider a presigned S3 URL returned by a cloud-storage API. The URL works only before its expiration time and only if the agent preserves its signature-bound query string byte-for-byte ( 3 ) . An agent can understand the user’s goal, call the right API, and still fail the workflow by using the URL too late, truncating it, re-encoding reserved characters, or reordering signed parameters. This failure pattern is not unique to presigned URLs. OAuth state parameters, session tokens, webhooks authenticated with HMAC (hash-based message authentication code), and rate-limit windows all impose the same two constraints: temporal validity (use the artifact before it expires or becomes stale) and byte-level integrity (relay the artifact without alteration). The two constraints are orthogonal. Refreshing an expired artifact repairs validity but does not prevent in-transit corruption of every fresh copy; preserving bytes intact does not teach the agent to respect deadlines or version freshness. Table 1 : Agent benchmark landscape. “V”: temporal validity; “I”: byte-level integrity; “P”: programmatic evaluation (no human or LLM judge). Benchmark V I P SWE-bench ( 22 ) ✗ ✗ ✓ ToolBench ( 46 ) ✗ ✗ Partial AgentBench ( 27 ) ✗ ✗ ✓ TravelPlanner ( 53 ) ✗ ✗ Partial \tau -bench ( 55 ) ✗ ✗ ✓ Terminal-Bench ( 30 ) ✗ ✗ ✓ TicToc ( 10 ) Partial ✗ ✓ ContractBench (Ours) ✓ ✓ ✓ Existing benchmarks partly address the temporal axis and largely overlook the integrity axis. Table 1 surveys the landscape: several include partial temporal constraints such as time limits or session windows, and recent work on temporal blindness shows that agents can mishandle stale information ( 10 ) . None, however, jointly tests temporal validity and byte-level artifact integrity. In practice, expired tokens, mutated URLs, stale ETags (HTTP entity tags), and signature mismatches break otherwise correct workflows. To address this gap, we introduce ContractBench , a benchmark for measuring observation-contract compliance in LLM agents. ContractBench contains 33 tasks drawn from real API contract patterns — presigned URLs, OAuth state parameters, signed requests, HMAC-protected webhooks, rate-limit windows, and multi-step token workflows — arranged across the two orthogonal axes and eight contract domains, shown in Figure 1 . Every task runs under a deterministic virtual clock, validates artifacts programmatically, and assigns each failed episode one primary label from a taxonomy of 15 failure modes that map directly onto either the validity or the integrity axis. The benchmark measures both whether an agent reaches a goal and whether it preserves the contracts attached to intermediate tool outputs. Across 38 model variants from 15 families, observation-contract compliance is unsolved at the frontier (best model 77.8 % on the full 33-task suite, n{=}99 ) and behaves non-monotonically with scale and post-training: a sharp within-family capability cliff in Qwen 3.5 and a V-shape across GPT-5 generations at near-constant parameter count. Our contributions are: 1. Observation contracts. We formalize observation contracts as tool-returned artifacts governed by two orthogonal constraints, temporal validity and byte-level integrity, and define a taxonomy of 15 failure modes for diagnosing contract failures. 2. ContractBench . We build a deterministic benchmark of 33 agent tasks drawn from real API contract patterns, covering validity-dominant, integrity-dominant, and dual-axis pressures. Each task has a programmatic validator, a reproducible virtual-clock execution model, and a reference solution that establishes solvability. 3. Empirical findings. Across 38 model variants spanning 15 families, contract compliance is neither guaranteed