The Verifier Tax: Horizon Dependent Safety–Success Tradeoffs in Tool Using LLM Agents Conference: ACM Conference on AI and Agentic Systems; 2026; TBD CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Natural language processing CCS: Computing methodologies Planning and scheduling CCS: Computing methodologies Machine learning

Tanmay Sah, Vishal Srivastava, Dolly Sah, Kayden Jordan · 2026

Read The Verifier Tax: Horizon Dependent Safety–Success Tradeoffs in Tool Using LLM Agents Conference: ACM Conference on AI and Agentic Systems; 2026; TBD CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Natural language processing CCS: Computing methodologies Planning and scheduling CCS: Computing methodologies Machine learning by Tanmay Sah, Vishal Srivastava, Dolly Sah, Kayden Jordan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19328.txt.

Introduction

Sah and co-authors measure the cost of guarding tool-using agents against unsafe acts. They test three agent designs on the tau-bench airline and retail tasks. Safety mediation blocks up to 94 percent of rule-breaking acts. Yet strictly safe task success stays below 5 percent in most settings. Many unsafe wins come from models inventing user IDs to skip login checks. After a block, agents rarely recover, with rates from 21 percent down to near zero. The team calls this gap a verifier tax on time and compute.

Catalog record

CREATORTanmay Sah, Vishal Srivastava, Dolly Sah, Kayden Jordan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58601
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19328

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Verifier Tax: Horizon Dependent Safety–Success Tradeoffs in Tool Using LLM Agents Conference: ACM Conference on AI and Agentic Systems; 2026; TBD CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Natural language processing CCS: Computing methodologies Planning and scheduling CCS: Computing methodologies Machine learning Tanmay Sah, Vishal Srivastava, Dolly Sah, Kayden Jordan Abstract . We study how runtime enforcement against unsafe actions affects end-to-end task performance in multi-step tool using large language model (LLM) agents. Using τ \tau -bench across Airline and Retail domains, we compare baseline Tool-Calling, planning-integrated ( Triad ), and policy-mediated ( Triad-Safety ) architectures with GPT-OSS-20B and GLM-4-9B. We identify model dependent interaction horizons (15–30 turns) and decompose outcomes into overall success rate (SR), safe success rate (SSR), and unsafe success rate (USR). Our results reveal a persistent “Safety-Capability Gap”. While safety mediation can intercept up to 94% of non-compliant actions, it rarely translates into strictly safe goal attainment (SSR < 5% in most settings). We find that high unsafe success rates are primarily driven by “Integrity Leaks,” where models hallucinate user identifiers to bypass mandatory authentication. Recovery rates following blocked actions are consistently low, ranging from 21% for GPT-OSS-20B in simpler procedural tasks to near 0% in complex Retail scenarios. These results demonstrate that runtime enforcement imposes a significant “verifier tax” on conversational length and compute cost without guaranteeing safe completion, highlighting the critical need for agents capable of grounded identity verification and post-intervention reasoning. 1. Introduction Large language models (LLMs) are increasingly deployed as autonomous agents that interact with external tools, APIs, and structured environments to complete user goals. Unlike single-turn assistants, agentic systems must sustain coherent decision making across sequential steps while handling partial observability, tool errors, and evolving user constraints. Prior work shows that integration of agents with external tools extend their capability from only text generation to searching, computing, and transacting. ( 25 ; 18 ; 8 ) . As such systems move toward higher-stakes applications, ensuring continuing effectiveness and compliance with explicit procedural constraints becomes critical. A common approach to enforcing safety in tool-using agents is verifier mediation which evaluates proposed actions or tool calls against predefined policies before execution. If a violation is detected during this runtime enforcement phase, the action is blocked and the agent must revise its plan ( 7 ; 2 ) . Such mechanisms are modular and can be layered onto existing agents without retraining. However, while runtime enforcement is intended to reduce unsafe behavior, its impact on end-to-end task outcomes and execution dynamics remains insufficiently characterized ( 27 ; 12 ; 3 ) . Most evaluations of interactive agents report aggregate task success, while safety violations are analyzed separately or in simplified settings that do not capture sustained multi-step interaction ( 28 ; 10 ) . In transactional tool-agent-user environments, \tau -bench provides explicit procedural policies and state-based evaluation, enabling systematic measurement of both task success and policy compliance ( 24 ) . This benchmark framework makes it possible to analyze not only whether unsafe actions are reduced, but how runtime enforcement reshapes the joint distribution of strictly safe success, unsafe success, and failure as interaction horizons increase. In this paper, we conduct a controlled empirical study of runtime safety mediation in tool-using LLM agents. We compare three architectures that differ in control-flow structure and enforcement design: direct Tool-Calling (baseline), a planning-based agent ( Triad ), and a policy-regulated variant ( Triad-Safety ). Using GPT-OSS-20B ( 15 ) and GLM-4-9B ( 6 ) , we evaluate these architectures on the Airline and Retail domains of \tau -bench, accounting for model-dependent interaction limits ( max_turns of 15 for GPT-OSS-20B and 30 for GLM-4-9B) to isolate model-specific conversational efficiency. We decompose outcomes into overall success rate (SR), safe success rate (SSR), and unsafe success rate (USR), and quantify verifier dynamics via intervention frequency and the recovery rate following blocked actions. Our results reveal a persistent “Safety-Capability Gap”. While safety-mediated architectures can intercept non-compliant actions in up to 94% of episodes, agents frequently fail to successfully revise their actions within the turn limit. Furthermore, we find that the majority of safety violations in high-performing agents are driven by “Integrity Leaks,” where models hallucinate user identifiers to bypass mandatory authentication steps. We characterize the resulting increase in conversational length and compute cost as a verifier tax and demonstrate that larger models, despite tighter turn constraints, exhibit superior conversational efficiency in safe goal attainment (GPT-OSS-20B saturates at SR@10–15, whereas GLM-4-9B requires up to 30 turns to converge). Our contributions include: (i) the decomposition of multi-step agent safety into safe vs. unsafe success, (ii) the identification of the Safety-Capability Gap in transactional environments, (iii) an analysis of the Verifier Tax on operational overhead (2.0–2.8 \times token inflation across all model \times domain settings), and (iv) the proposal of Strict Grounding Gates to mitigate integrity leaks. 2. Related Work Safety guardrails and runtime enforcement. Safety mechanisms for LLM systems are commonly implemented through guardrails that constrain model outputs or actions. However, guardrails often fail in transactional settings because they operate as static lexical filters that lack the procedural state-awareness needed to verify complex multi-turn dependencies (e.g., ensuring a user provided a specific ID earlier in the session). Another approach to safety is alignment, such as Constitutional AI, which shapes model behavior through rule-guided feedback ( 2 ) . However, these approaches are prone to the “Safety-Capability Gap,” where alignment can degrade under long-horizon pressure or lead to over-refusals that halt task progress. Moderation systems typically classify or filter unsafe outputs post-hoc ( 7 ) . In contrast, we evaluate runtime verifier mediation , which provides a modular architectural layer for proactive runtime enforcement allowing the system to identify and block unsafe tool calls before they are executed in the environment, triggering a “Block-and-Revise” loop that we use to quantify the resulting Verifier Tax. MIRAGE-Bench ( 26 ) highlights agentic hallucinations, including tool-parameter fabrication, which resonates with our “Integrity Leaks” category. Architectural alternatives such as CE-MCP ( 5 ) suggest code-level gating and sandboxed validation, providing a deterministic alternative to the heuristic textual verifiers we evaluate here. Intervention and corrective loops. Intervention mechanisms that revise intermediate reasoning or plans have been explored to improve task performance through episodic self-feedback ( 19 ) and iterative self-correction ( 11 ) . More recently, critique-guided frameworks have been proposed to improve actor outputs by providing specific, instruction based feedback on errors ( 23 ) . Our ’Block-and-Revise’ loop functions as a specific application of these frameworks where the verifier acts as a safety critical monitor. Specifically, the verifier’s rejection signal provides a targeted critique that forces the agent to align its next action with procedural policies. We extend these insights by isolating the unique operational overhead (the verifier tax) and the actual safety efficacy of such corrective loops in policy-constrained transactional tasks. Long-horizon reliability. Long-horizon environments introduce compounding challenges for recovery and robustness. While prior work on structured planning focuses on task decomposition ( 14 ; 1 ) , our study specifically addresses reliability by quantifying an agent’s ability to maintain progress following policy-driven interventions. By decomposing outcomes into safe and unsafe success, we expose cases where agents achieve reliability only by taking non-compliant shortcuts

Read the complete text