Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents

Yufeng Wang · 2026

Read Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents by Yufeng Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00476.txt.

Introduction

Wang asks if chat agents do what they say, or just what they claim to think. Faith in an agent's acts rests on that match. No gold act exists in most fields to check it by. Wang builds a test bed where each call has a known right act.

A Texas poker sim gives that gold frame for the test. The gap is split in two steps for a clean read. Step one checks if a stated call tracks the agent's own thought. Step two checks if the act tracks the stated call. Step two proves firm when calls are read from a clear tag. Free-text reads swell the gap with noise. Step one frays in split ways, not one mode. Rule slips lean to the safe side near all the time.

Catalog record

CREATORYufeng Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00476

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents Yufeng Wang Abstract Do LLM agents act on the reasoning they state? This question of process fidelity is central to using LLMs in social simulation, yet it is hard to measure where no reference for correct behavior exists. We study it in a controlled setting—a Texas Hold’em simulator with a verifiable reference action for every decision—by decomposing the faithfulness gap into two steps: reasoning → \rightarrow conclusion (does the stated decision follow from the agent’s own reasoning?) and conclusion → \rightarrow action (does the agent execute what it states?). The two steps behave oppositely. Conclusion → \rightarrow action is reliable: inconsistency is 0.0–1.8% across three model families, including a natively trained reasoning model, and the 22–26% reported by free-text conclusion extraction is largely a measurement artifact. The real gap is upstream—in 65% of erroneous decisions the agent estimates the inputs correctly and restates the rule, then draws a conclusion that contradicts it, and injecting the rule into the prompt does not help. Process-fidelity evaluation should therefore target the reasoning → \rightarrow conclusion step and elicit machine-checkable conclusions to avoid conflating measurement noise with model behavior. We release code and a live multi-agent demo. 1 Introduction A fundamental concern in LLM-based social simulation is whether agents genuinely act on the reasoning they appear to exhibit, or produce surface-level rationalizations that do not govern behavior ( 4 ; 9 ) . This concern is sharpened by a growing critique that the field deploys simulators faster than it validates them, and that reported results depend heavily on the elicitation protocol and measurement instrument used ( 5 ) . The question is difficult to resolve in open social settings because ground truth is absent: no objective criterion exists for the correct action in a negotiation or a cultural exchange. Competitive games with a computable reference policy offer a different regime. In Texas Hold’em poker, a reasonable reference action at each decision point can be derived from hand equity and pot odds—a fixed reference that is unavailable in most social simulation contexts. This property renders poker a controlled calibration environment for process fidelity : when an LLM agent claims to reason about pot odds and hand strength, the executed action, the stated conclusion, and the intermediate reasoning can each be inspected against one another and against the reference. Open-ended social simulations—marketplaces, negotiations, multi-agent societies—lack this verifiability, yet the agents that populate them are the same models, prompted in the same way. A failure mode that is measurable in poker is therefore a candidate mechanism for the harder-to-measure breakdowns of persona consistency and emergent behavior reported in richer settings such as SOTOPIA ( 9 ) . Using this setup, we decompose the “faithfulness gap” into two distinct steps and find that they behave very differently. The first step, conclusion \rightarrow action , asks whether the agent executes the decision it states. The second step, reasoning \rightarrow conclusion , asks whether the stated decision follows validly from the agent’s own intermediate reasoning. Prior work has reported that CoT explanations are frequently unfaithful ( 6 ) and that LLMs exhibit a “knowing-doing gap” in game-theoretic settings ( 3 ) ; our results refine this picture by showing that the gap is concentrated almost entirely in the second step. We study process fidelity in LLM agents using poker as a controlled strategic environment with observable intermediate reasoning and measurable reference decisions. Across three LLM families (Claude Haiku 4.5, Gemini 2.5 Flash-Lite, and DeepSeek-v4-pro—the last a natively trained reasoning model) and four prompt strategies, we address two research questions: • RQ1 : How large is the conclusion \rightarrow action inconsistency rate under game-theoretic pressure, and how sensitive is the measured rate to how the conclusion is elicited and parsed? • RQ2 : When the reasoning is made explicit, do agents derive conclusions that follow from their own stated inputs and rules; and does injecting the decision rule into the prompt improve adherence to it? Contributions. • A measurement caution for CoT faithfulness. The measured stated-vs.-actual inconsistency rate is an order of magnitude lower under explicit-conclusion elicitation (below 2%) than under free-text phrase extraction (22–26%), indicating that a substantial part of reported CoT “unfaithfulness” can reflect conclusion-extraction noise rather than model behavior. • A decomposition that localizes the gap. Separating reasoning \rightarrow conclusion from conclusion \rightarrow action shows the faithfulness gap is concentrated upstream: agents execute their stated decisions almost perfectly, but derive those decisions invalidly from their own stated reasoning. • A characterization of the upstream failure. The dominant error (65% of failures) is rule misapplication : agents estimate inputs correctly and restate the rule, then override it with qualitative hedging in a systematically risk-averse direction. Injecting the rule into the prompt does not help, isolating rule application —not retrieval—as the bottleneck. • A portable process-fidelity instrument. We package poker as a verifiable, reproducible testbed with a step-level fidelity probe (open harness and live multi-agent demo) that can be ported to richer social simulations to detect persona drift and surface-level rationalization before they propagate. Figure 1: Overview of the experimental framework. (A) Four LLM agents occupy distinct strategy seats at a Texas Hold’em table; an equity-based threshold heuristic provides a reproducible reference action for every decision. (B) The decision pipeline decomposes into two steps: reasoning \rightarrow conclusion (does the stated decision follow from the agent’s own inputs and rule?) and conclusion \rightarrow action (does the executed action match the stated decision?). (C) The measured inconsistency lives almost entirely in the upstream step: conclusion \rightarrow action inconsistency is near zero once the conclusion is elicited explicitly, while reasoning \rightarrow conclusion errors are frequent. The answers bear directly on the central concern of this workshop: distinguishing genuine strategic behavior from model artifacts. We find that apparent stated-vs.-actual unfaithfulness is, in this setting, sensitive to how the conclusion is measured, while a less commonly measured failure—invalid reasoning that the agent nonetheless executes faithfully— is both real and substantial. 2 Related Work Chain-of-thought faithfulness. 7 demonstrated that chain-of-thought prompting substantially improves LLM performance on multi-step reasoning tasks. 6 showed that CoT explanations can be unfaithful— models influenced by spurious biasing features fail to mention those influences while still acting on them—a finding established through careful controlled manipulation rather than free-text conclusion extraction. Our results complement this line of work with a methodological observation specific to our setting: when the stated conclusion is inferred from free-text reasoning rather than controlled directly, the measured stated-vs.-actual inconsistency rate is highly sensitive to the extraction method, varying by an order of magnitude between a phrase parser and an explicit decision tag. This concerns how the gap is operationalized, not whether CoT can be unfaithful. LLM multi-agent social simulation. 4 showed that LLM agents equipped with memory, reflection, and planning can produce emergent social behaviors in sandbox environments. SOTOPIA ( 9 ) introduced an interactive benchmark for social intelligence, finding that GPT-4 achieves significantly lower goal-completion rates than humans on challenging social scenarios. 5 survey this fast-growing area and argue that it must pivot from expansion to consolidation around reproducible, validated evaluation—calling for operational validity , the property that a simulator reproduces rather than

Read the complete text