Assertion-Conditioned Compliance: A Provenance-Aware Vulnerability in Multi-Turn Tool-Calling Agents

Daud Waqas, Aaryamaan Golthi, Erika Hayashida, Huanzhi Mao · 2025

Read Assertion-Conditioned Compliance: A Provenance-Aware Vulnerability in Multi-Turn Tool-Calling Agents by Daud Waqas, Aaryamaan Golthi, Erika Hayashida, Huanzhi Mao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00332.txt.

Introduction

AI assistants that call external tools across several turns can act on the world, which raises safety stakes. Daud Waqas, Aaryamaan Golthi, Erika Hayashida, and Huanzhi Mao study a vulnerability in such agents in 2025, which they call assertion-conditioned compliance. The agents serve business, medical, and financial operations.

Multi-turn pipelines remain hard to deploy in safety-critical industries because model resilience is still uncertain. A provenance-aware vulnerability means the agent's trust depends on where earlier claims in the conversation came from. Exposing how asserted statements can steer tool use helps designers harden assistants before they handle sensitive work. Provenance, knowing where a claim entered the conversation, is the defense the title highlights.

Catalog record

CREATORDaud Waqas, Aaryamaan Golthi, Erika Hayashida, Huanzhi Mao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00332

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Assertion-Conditioned Compliance: A Provenance-Aware Vulnerability in Multi-Turn Tool-Calling Agents Daud Waqas, Aaryamaan Golthi, Erika Hayashida, Huanzhi Mao Abstract Multi-turn tool-calling LLMs — models capable of invoking external APIs or tools across several user turns — have emerged as a key feature in modern AI assistants, enabling extended dialogues from benign tasks to critical business, medical, and financial operations. Yet implementing multi-turn pipelines remains difficult for many safety-critical industries due to ongoing concerns regarding model resilience. While standardized benchmarks, such as the Berkeley Function-Calling Leaderboard (BFCL), have underpinned confidence concerning advanced function-calling models (like Salesforce’s xLAM V2), there is still a lack of visibility into multi-turn conversation-level robustness, especially given their exposure to real-world systems. In this paper, we introduce Assertion-Conditioned Compliance (A-CC) , a novel evaluation paradigm for multi-turn function-calling dialogues. A-CC provides holistic metrics that evaluate a model’s behavior when confronted with misleading assertions originating from two distinct vectors: (1) user-sourced assertions (USAs), which measure sycophancy toward plausible but misinformed user beliefs, and (2) function-sourced assertions (FSAs), which measure compliance with plausible but contradictory system policies (e.g., stale hints from unmaintained tools). Our results show that models are highly vulnerable to both USA sycophancy and FSA policy conflicts, confirming A-CC as a critical, latent vulnerability in deployed agents. 1 Introduction † † footnotetext: Code: GitHub Large Language Models (LLMs) augmented with the ability to invoke external tools (tool-calling) have demonstrated remarkable capabilities beyond their standalone performance 6 ; 13 ; 16 . However, their successful deployment in safety-critical industries remains constrained by concerns about transparency and the evaluation of their reasoning robustness 7 . When such agents encounter incorrect or contradictory information — an event familiar in real-world user interactions with LLMs 5 , and a serious concern in unmaintained deployments — an agent that treats this feedback as authoritative can propagate incorrect states and cause harmful downstream degradation in both the tool-calling pipeline and the real-world systems it influences 20 . Previous work has documented linguistic sycophancy 15 ; 4 and has confirmed that tool-augmented agents are vulnerable to safety issues arising from both misleading user inputs and contradictory tool data 21 . However, the provenance of these misleading signals—and how an agent’s compliance bias differs when the source is the User (a social cue) versus the Function (a system cue)—remains poorly understood and has yet to be systematically evaluated. Figure 1: Overview of baseline, user-sourced assertion (USA), and function-sourced assertion (FSA) behavior in multi-turn tool-calling. Assertions may cause compliance in the immediate function call, after which the model may either propagate the misleading path (degradation) or recover and produce the correct final environment state. We introduce Assertion-Conditioned Compliance (A-CC) to formally diagnose this gap. A-CC is a failure mode in which model accepts an incorrect assertion and updates its internal reasoning to incorporate it, thus manipulating its execution pipeline. This goes beyond standard linguistic sycophancy, which is limited to verbal agreement in the surface response. A-CC instead captures procedural compliance, in which the execution pipeline itself is altered. In this setup, provenance is central. We measure compliance across two distinct vectors: plausible but incorrect user-sourced assertions (USAs), which test sycophancy’s effect on tool-use, and contradictory function-sourced assertions (FSAs), which test whether the agent allows erroneous tool feedback to redirect its execution pipeline. This latter FSA vector models a plausible and insidious industrial risk (where, for example, stale hints may originate from unmaintained tools in mismanaged deployments; while unlikely, this would have a global effect across said deployments) (illustrated in Figure 1 ). We ground our evaluation in the BFCL 11 , extending its multi-turn, state-dependent tasks from a simple accuracy metric to a diagnostic of procedural robustness in realistic tool APIs. Our experiments show how A-CC exposes a consistent behavioral weakness across model families and sizes: plausible assertions reliably reshape tool-calling pipelines, even when final BFCL accuracy remains high. Importantly, we find that assertion compliance is not tightly coupled to accuracy degradation, indicating that assertion-following carries risks beyond the degradation of the user’s task. For industry settings, this means that models judged solely on task success accuracy by leaderboard scores may still execute unnecessary or unsafe operations under natural user prompt variation or stale tool hints. By making provenance and procedural compliance explicit, A-CC provides a practical diagnostic that reinforces the need for additional guardrails and safety measures. In summary, our contributions are as follows: • A new evaluation paradigm (A-CC) that formalizes and measures how large language models incorporate incorrect assertions into their reasoning and tool-calling pipelines. • Two complementary assertion sources — user-sourced and function-sourced — that expose distinct behavioral risks: social compliance to user cues and procedural compliance to internal system feedback. • A reproducible benchmarking framework for empirically comparing behavioral patterns across model families. • A compliance-based evaluation metric (compliance rate) that decouples obedience to asserted operations from task success, quantifying how often agents execute unnecessary or unsafe functions even when pipelines are otherwise correct. 2 Related Work Sycophancy in Large LMs. Prior work has shown that LLMs trained via RLHF often produce outputs that align with a user’s expressed belief or preferences, even when doing so sacrifices accuracy or validity 15 . 18 extend this line of work by quantifying sycophancy across opinion and arithmetic-based settings, showing that a lightweight synthetic-data intervention can reduce it. More recently, 8 move beyond single-turn prompts and introduce a benchmark for evaluating multi-turn sycophancy, demonstrating that conformity to user beliefs can intensify over extended dialogue. Our A-CC framework is complementary: rather than focusing on agreement in surface text, we study how such assertion-following manifests into procedural compliance in the function-calling pipeline of tool-augmented agents. Benchmarks for function-calling. Our study builds on the multi-turn tasks of the BFCL 11 , using its real-world APIs and final-environment-state scoring as a natural substrate for analyzing assertion-conditioned robustness. BFCL’s multi-turn category couples stateful tool interactions with a fixed success metric, which allows us to isolate how provenance-tagged assertions reshape the execution pipeline without modifying the underlying benchmark. Other function-calling benchmarks are complementary but target different aspects of the problem. Robustness and adversarial contexts in tool use. A growing strand of research examines failures when the “environment around” the agent shifts or is hostile. Prior work shows that naturalistic prompt variation, toolkit expansion with semantically similar tools, and distributional shifts in tool selection all induce agentic degradation and can collapse certified worst-case accuracy under adversarial tool perturbations ( 14 ; 22 ) . Other work also demonstrates how deceptive or malicious tool injections can trigger privacy leaks, DoS actions, or unintended executions, and that tool-calling modes expose jailbreak paths not visible in standard chat interfaces ( 24 ) . Stage-based safety suites such as ToolSword 21 stress-tests input, execution, and output stages of tool-calling LLMs under explicitly specified safety scenarios, measuring attack success, unsafe tool selection, and harmful or erroneous feedback propagation; our work is complementary in that we study how non-adversarial, provenance-tagged assertions within standard benchmarks reshape procedural compliance, rather than curating new safety scenarios. Alignment and decision-making in tool use. Beyond defenses, alignment-oriented work specifies desirable behaviors. 3 proposes H2A—Helpfulness, Harmlessness, Autonomy—and releases ToolAlign to train models that (i) call tools to help, (ii) refuse unsafe instructions and insecure

Read the complete text