The Point of No Return: Counterfactual Localization of Deceptive Commitment in Language-Model Reasoning

Scott Merrill, Shashank Srivastava · 2026

Read The Point of No Return: Counterfactual Localization of Deceptive Commitment in Language-Model Reasoning by Scott Merrill, Shashank Srivastava (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17113.txt.

Introduction

Merrill and Srivastava ask when a language model locks into deception during its own reasoning, rather than judging only its final answer. They test sentence prefixes across five staged settings, resampling continuations to locate commitment points, and build a corpus of about 1.46 million sentences. Attention patterns, they report, predict those turning points across settings better than word cues do.

Catalog record

CREATORScott Merrill, Shashank Srivastava
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE127231
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17113

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Point of No Return: Counterfactual Localization of Deceptive Commitment in Language-Model Reasoning Scott Merrill, Shashank Srivastava Abstract Existing deception datasets label completed outputs as honest or deceptive, treating deception as a property of the final response rather than a function of the model’s reasoning trace . This obscures a more fundamental question: when does a language model become committed to deception? We introduce counterfactual localization : for each sentence prefix in a reasoning trace, we fix the prefix, resample continuations, and estimate the probability of a deceptive outcome. To scale this, we construct five environments (spanning strategic bluffing, maze guidance, financial advice, used-car sales, and offer negotiation) in which deception is never prompted but emerges from strategic incentives and labels follow mechanically from environment state rather than subjective human judgment. The resulting corpus localizes ∼ \sim 1.46M sentences across four reasoning models, drawn from over 94.1M sampled continuations, 91.5B generated tokens, and over 100K scenarios. Sentence-level human evaluation confirms that detected commitment points correspond to interpretable shifts in decision state. Using this resource, we show that lexical cues for commitment prediction transfer poorly across environments, whereas attention-based transition features generalize out of distribution, suggesting that deceptive commitment is reflected in reusable changes in reasoning dynamics rather than surface form. We further identify compact attention-head sets (under 10% of heads) that, selected on one environment, causally suppress deceptive commitment across held-out environments. We release the corpus as a substrate for studying deception, and more broadly commitment, in language-model reasoning. 1 Introduction Large language models can produce strategically misleading responses [ 31 , 11 , 12 ] . Yet most existing deception datasets label completed outputs as honest or deceptive, treating deception as a property of the final response [ 22 , 15 , 28 ] . This framing obscures a more fundamental question: when does a language model become committed to deception within its reasoning trace? It cannot tell us which intermediate reasoning steps make deception likely, whether those signals transfer across settings, or which internal mechanisms causally support deceptive commitment. We argue that understanding deception requires modeling it as a dynamic function of partial reasoning , rather than as a label on a completed output [ 17 ] . A partially generated trace can support multiple futures: some continuations remain honest, while others become deceptive. As more of the trace is fixed, the probability of deception can shift gradually or abruptly, revealing points of deceptive commitment where the model becomes substantially more likely to complete the trajectory deceptively. To study this process, we introduce counterfactual localization : for each sentence prefix in a reasoning trace, we fix the trace through that prefix, sample many continuations, and estimate the resulting deception rate. A sentence matters not merely because it appears in a deceptive trajectory, but because fixing it changes the distribution over future deceptive continuations. We call sharp changes in this rate commitment junctures . The key to scaling counterfactual localization is intrinsic supervision. Prior deception datasets often rely on human-written, human-labeled, or human-validated examples, and human deception judgments are known to be noisy [ 22 , 3 ] . We instead construct five environments (focused on strategic bluffing, maze guidance, financial advice, used-car sales, and offer negotiation) in which deception arises from strategic incentives and labels follow mechanically from environment state rather than subjective human judgment. Across these environments, deception takes qualitatively different forms, including explicit false claims, misleading guidance, self-serving recommendation, selective concealment, and bargaining misrepresentation. The environments also differ in action space, observability structure, incentives, and language form, providing a difficult testbed for identifying which localized signals of commitment transfer across settings rather than reflecting environment-specific artifacts [ 7 , 14 ] . Combining counterfactual localization with intrinsic supervision, we construct, to our knowledge, the largest deception dataset by token count. 1 1 1 https://huggingface.co/datasets/anonymous-neurips-2026-ED/deception-localization The corpus contains approximately 1.46M localized sentences across 100K scenarios and four reasoning models, derived from \sim 5.3B sampled continuation sentences, \sim 91.5B generated tokens, and 2.22 TB of trace and continuation data. Sentence-level human evaluation confirms that detected commitment points correspond to interpretable shifts in decision state. Using this corpus, we train predictors from lexical, activation-based, and attention-based features and evaluate them under leave-one-environment-out transfer. We then ask whether predictive signals correspond to causal mechanisms: using attribution patching [ 32 , 34 ] , we identify compact attention-head circuits whose sentence-level patching suppresses deceptive commitment both in-domain and across held-out environments. Our contributions are: 1. We reframe deception detection from binary output classification to modeling deception as a function of the reasoning trace , and introduce counterfactual localization to estimate sentence- and prefix-level deception rates from sampled continuations. 2. We construct five deception environments with intrinsic, mechanically derived labels in which deception emerges from strategic incentives rather than instruction. The resulting corpus consists of 1.46 M localized sentences across four reasoning models, \sim 91.5 B generated tokens, and 2.22 TB of trace and continuation data. This is, to our knowledge, the largest deception dataset, and is validated by sentence-level human annotation. 3. We show that lexical cues transfer poorly across environments, whereas attention-based transition features generalize out of distribution , suggesting that deceptive commitment is reflected in reusable changes in reasoning dynamics rather than stable surface patterns. 4. Across all the reasoning models evaluated, we identify a compact attention-head circuit (under 10% of heads) whose patching causally suppresses deceptive commitment in-domain and across held-out environments, providing evidence that commitment signals are not only predictive but also mechanistically manipulable. 2 Related Work Deception Detection in Language Models. Language models can misrepresent their situation, manipulate, and exploit users to achieve goals [ 11 , 20 , 5 ] . Prior work builds supervised detectors using lexical, syntactic, neural, and cross-corpus features across opinion spam, dialogue, games, and LLM deception [ 19 , 23 , 6 , 29 , 33 , 28 , 15 , 31 , 26 ] , or detects lying through follow-up probing and instructed honesty/deception contrasts [ 24 , 15 ] . These approaches share two limitations: they assign a single label to a completed output, often from constructed or noisy human-judged examples [ 23 , 27 , 3 ] , and deception is usually prompted . This framing also degrades under domain shift [ 25 , 8 , 33 ] , partly because output labels compress reasoning into a single trajectory and discard internal signals that precede deception. We instead design environments where deception emerges from strategic incentives, letting us label counterfactual continuations from the same prefix without manual judgments, and test if signals transfer across environments [ 7 , 14 ] . Localizing and Intervening on Reasoning. Process supervision argues that intermediate reasoning steps can be more informative than final answers [ 17 ] , and recent counterfactual sampling work identifies thought anchors , sentences that disproportionately shape downstream reasoning [ 2 ] . A parallel line probes activations for truthfulness or deception, finding linear structure that separates true from false statements [ 1 , 18 , 9 ] , but assesses completed statements from a static activation snapshot. Mechanistic interpretability probes causal roles via causal mediation, activation and attribution patching, and circuit discovery [ 34 , 32

Read the complete text