Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety

Xuan Chen, Lu Yan, Ruqi Zhang, Xiangyu Zhang · 2026

Read Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety by Xuan Chen, Lu Yan, Ruqi Zhang, Xiangyu Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18245.txt.

Introduction

Chen and co-authors present SafeAudit, a meta-audit framework for LLM agent tool-call safety. Benchmarks test many settings, but their coverage of unsafe tool patterns is unclear. An LLM enumerator builds test cases by listing valid tool workflows and user scenarios. A rule-resistance metric distills compact rules from benchmarks and finds gaps under them. Across 3 benchmarks and 12 environments, over 20% residual unsafe behaviors surfaced.

Catalog record

CREATORXuan Chen, Lu Yan, Ruqi Zhang, Xiangyu Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61966
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18245

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety Xuan Chen, Lu Yan, Ruqi Zhang, Xiangyu Zhang Abstract Large Language Model (LLM) agents increasingly act through external tools, making their safety contingent on tool-call workflows rather than text generation alone. While recent benchmarks evaluate agents across diverse environments and risk categories, a fundamental question remains unanswered: how complete are existing test suites, and what unsafe interaction patterns persist even after an agent passes the benchmark? We propose SafeAudit , a meta-audit framework that addresses this gap through two contributions. First, an LLM-based enumerator that systematically generates test cases by enumerating valid tool-call workflows and diverse user scenarios. Second, we introduce rule-resistance, a non-semantic, quantitative metric that distills compact safety rules from existing benchmarks and identifies unsafe interaction patterns that remain uncovered under those rules. Across 3 benchmarks and 12 environments, SafeAudit uncovers more than 20% residual unsafe behaviors that existing benchmarks fail to expose, with coverage growing monotonically as the testing budget increases. Our results highlight significant completeness gaps in current safety evaluation and motivate meta-auditing as a necessary complement to benchmark-based agent safety testing. 1 Introduction LLM agent systems combine natural language reasoning with tool use such as email, calendar, and web actions, and are rapidly moving from research prototypes to widely deployed products ( 50 ; 7 ; 27 ) . As these agents gain access to tools with real-world side effects, evaluating and improving their safety becomes a first-order concern: a single unsafe tool invocation can cause financial loss or irreversible changes ( 13 ; 8 ) . A growing body of work has proposed safety benchmarks for LLM agents by designing diverse sandbox environments, toolsets, and user intents ( 25 ; 36 ; 51 ) , standardizing evaluation, and exposing many failure cases. As shown in Figure 1 , existing benchmarks are often curated through helper LLM generation and expert review ( 54 ; 22 ) . Although they are designed to cover various environments and tools, an important question remains: if an agent passes a benchmark in a sandbox environment, to what extent can we claim the agent is “safe”? Figure 1: Existing benchmarks mainly focus on the diversity of environments and tools. SafeAudit discovers blind spots that benchmarks fail to cover. We address a fundamental completeness gap in current agent safety evaluation. Existing benchmarks emphasize coverage of environments and tools, and are not designed to systematically characterize the space of unsafe behavioral patterns, nor to quantify how much of that space has been tested. Even with the same environment and tools, small changes in the user scenario can lead to qualitatively different unsafe outcomes ( 26 ; 28 ) . For example, ambiguity in tool API parameters 9 , or instruction conflicts ( 60 ; 37 ) can each flip an agent from requesting verification to executing an unsafe action. As such, validating an agent on one scenario is insufficient to guarantee safety under other scenarios that the benchmark does not enumerate. Motivated by this gap, we propose SafeAudit , a meta-audit framework that discovers blind spots of existing benchmarks and evaluates their completeness. Rather than proposing another benchmark, we ask: given an environment and its tool set, what is the residual space of scenarios in which an agent can still behave unsafely under the guardrails learned from existing benchmarks? Answering this requires overcoming two challenges. First, the space of valid test cases is combinatorially large: each case must specify a user request consistent with the environment’s constraints while remaining realistic and capable of eliciting unsafe behavior ( 18 ; 34 ) . Second, completeness is not directly observable; the full space of unsafe scenarios is unbounded, and semantic similarity between unsafe behaviors is an unreliable proxy for whether two cases expose the same underlying vulnerability ( 33 ) . To address these challenges, SafeAudit introduces an LLM-based enumerator that systematically generates executable agent test cases by enumerating valid tool-call chains and diverse user scenarios under that specific environment. To quantify approximate completeness, our key idea is rule-resistance: we distill compact safety rules from an existing benchmark and use them to protect the target agent, then the scenarios where the agent still produces unsafe tool-call trajectories under these rules are considered as uncovered unsafe behaviors. Empirically, compared to baselines, SafeAudit uncovers more than 20% residual unsafe cases that existing benchmarks fail to expose, revealing generalization gaps that suggest benchmark pass rates alone can overestimate real-world safety. In summary, our contributions are: • We design an LLM-based enumerator that systematically explores the space of valid agent workflows by generating executable tool-call sequences and the corresponding user scenarios, enabling broad-coverage test case generation within a given environment and tool set. • We introduce rule-resistance, a non-semantic and verifiable evaluation protocol that measures the novelty of test cases under benchmark-derived guardrails. This yields a relative signal of completeness grounded in tool-call traces rather than subjective interpretation. We show that rules do not degrade utility and introduce over-refusal, validating its practicality as a proxy for coverage. • We analyze the unsafe interaction patterns elicited by SafeAudit , and identify \sim 11\% novel patterns not covered by existing test suites under the same tools and environments. The meta-auditing of 3 agent-safety benchmarks across 12 environments with 8 backbone LLMs finds that existing benchmarks systematically leave more than 20% of unsafe interaction patterns uncovered. 2 Related Work 2.1 LLM Agent Safety Benchmarks Early benchmarks focus on high-level safety properties: R-Judge ( 53 ) evaluates whether LLMs can identify safety risks in agent interaction records, while other works ( 64 ; 48 ; 19 ) provide sandboxed simulation environments to test safety across diverse scenarios. A range of benchmarks target harmful agent behavior more directly ( 58 ; 59 ; 2 ; 14 ) , covering a range of harmful tool uses and adversarial user intents. More recent works ( 36 ; 45 ; 20 ) focus on real-world tools and consequential, multi-turn interactions with the user. Beyond harm, 30 assess awareness of privacy norms in agent actions. At the tool-call level, 23 propose step-level guardrails and feedback mechanisms for detecting unsafe tool invocations. 15 evaluate agents against long-horizon attacks, and 4 systematically analyze failure modes in multi-agent LLM systems. Despite this breadth, existing benchmarks share a common limitation: test case generation via prompting tends to surface obvious and well-anticipated failure patterns while leaving valid but rare interaction scenarios untested. There is no systematic method to enumerate the space of possible unsafe interactions, and no principled way to evaluate the coverage of different test sets. Without a structured map of the input space, passing an existing benchmark does not guarantee that an agent has been audited against the full range of risks. 2.2 Automated Testing and Red-Teaming Automated red-teaming has been widely studied in the context of specific attack vectors. Prior works benchmark agents’ robustness against prompt injection ( 56 ; 5 ) , discover prompt injection vulnerabilities ( 40 ; 38 ) , and its defenses ( 55 ; 31 ) . For jailbreaking, 11 and 52 exploit adversarial inputs to compromise multimodal agent networks, while others target web agents ( 44 ; 21

Read the complete text