Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning

Swapnil Parekh · 2026

Read Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning by Swapnil Parekh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11467.txt.

Introduction

Swapnil Parekh and Naman Goyal introduce ProFIL, a probe-filtered reinforcement learning extension that cuts reasoning theater in chain-of-thought models. Reasoning models often rationalize answers already chosen internally, producing deliberative-looking steps that add nothing to correctness. A probe trained once on the frozen base model spots post-commitment steps from internal activations. During GRPO, rollouts scoring above a threshold have their advantage zeroed. Across four reasoning domains, ProFIL cuts theater by eleven to one hundred percent and shortens chains while preserving accuracy.

Catalog record

CREATORSwapnil Parekh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62724
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11467

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning Swapnil Parekh Abstract Reasoning models post-hoc rationalize answers they have already committed to internally, producing chains of reasoning theater : deliberative-looking steps that contribute nothing to correctness. This wastes inference tokens, pollutes interpretability, and obscures what the model actually computed. We introduce ProFIL ( Pro be- Fil tered Reinforcement Learning) to reduce theater, increase chain-of-thought faithfulness, and shrink chain length in a single, drop-in extension to Group Relative Policy Optimization (GRPO). A multi-head attention probe is trained once on the frozen base model to detect post-commitment steps from internal activations alone; during GRPO, rollouts whose probe score exceeds a threshold have their advantage zeroed. Our central finding is that a probe trained on a frozen base, with verifier-derived labels and no human annotation, provides a stable signal that suppresses theater while resisting the RL-obfuscation failure mode predicted by prior work. Across four reasoning domains (GSM8K, LiveCodeBench, ToolUse, MMLU-Redux) and two model architectures (Llama-8B, Qwen-7B), ProFIL reduces post-commitment theater by 11–100% , raises faithful-fraction (e.g., + 24 +24 pp on LiveCodeBench under an independent Claude 3.7 Sonnet judge), and shortens chains by 4 4 – 19 % 19\% , all while preserving or improving task accuracy. ProFIL also beats a matched length-penalty GRPO baseline, isolating the gain as semantic commitment-detection rather than chain compression. Probe weights, training configurations, and rollouts are released across all four domains. 1 Introduction Figure 1: Reasoning theater before and after ProFIL (perf-ratio 0.60\rightarrow 0.00 , 6 post-commit steps eliminated). A randomly-sampled high-performativity GSM8K rollout (performativity ratio >0.9 ) from DeepSeek-R1-Distill-Llama-8B under standard GRPO (left) and the paired rollout under ProFIL (right) on the same question. Each row is one reasoning step; color encodes the frozen-base probe’s per-step performativity score (green = faithful, red = performative). The baseline commits to the correct answer at step 5 but continues for six more post-hoc elaboration steps; ProFIL halts at the dashed line marking the commitment point. Both models produce the correct answer. While scaling up reasoning with reinforcement learning has produced remarkable results [ 6 , 22 , 12 ] , a converging body of evidence reveals a systematic gap between what these models appear to do and what they actually do. Biased prompts steer final answers while leaving the chain of thought unchanged [ 20 ] ; models recover correct answers from truncated chains far earlier than their length implies [ 14 ] ; large-scale audits of production reasoning models find post-hoc rationalization to be widespread [ 4 ] . The pattern is consistent: reasoning models commit to an answer internally, then continue generating steps that look deliberative but contribute nothing. We call this reasoning theater (Figure 1 ). Theater carries a real cost. Post-commitment steps pollute interpretability, introduce spurious signal into process-reward training [ 21 , 16 ] , and inflate token cost in agentic deployments where inference budget matters. Prior work has measured this gap carefully but largely treated it as a diagnostic. Our goal is to reduce it : to suppress theater, raise faithfulness, and shrink chains for inference efficiency, while preserving or improving task accuracy. Faithfulness is non-verifiable: unlike answer correctness, no ground-truth label is available at the end of a rollout, and standard GRPO reward shaping cannot reach behaviors the verifier never sees. Our key observation is that the verifier we need is already inside the model. We introduce ProFIL (Probe-Filtered Reinforcement Learning), which builds an annotation-free faithfulness signal from internal activations (Figure 2 ): a multi-head attention probe is trained once on the frozen base model to detect post-commitment steps, and during GRPO any rollout whose mean probe score exceeds a threshold has its advantage zeroed. Because the probe runs on the frozen base rather than on the policy being trained, its decision boundary stays stable throughout RL. Contributions. • Goal-aligned method. A single, drop-in extension to GRPO that targets the three things that make theater costly: it suppresses post-commitment elaboration, raises faithful-fraction, and shrinks chain length, while preserving or improving task accuracy. • Generality. Consistent gains across four reasoning domains (GSM8K, LiveCodeBench, ToolUse, MMLU-Redux) and two model architectures (Llama-8B, Qwen-7B), robust to a matched length-penalty ablation and confirmed by an independent Claude 3.7 Sonnet judge. • Anti-gaming. The frozen probe attains AUROC 1.000 on LiveCodeBench rollouts before, during, and after training, addressing the RL-obfuscation failure mode of 9 : the model learns to suppress post-commitment steps, not to hide them from the probe. • Mechanism. Theater is a separable bimodal mode with a concrete linguistic signature (structural re-derivation blocks); ProFIL collapses the high-theater mode at all chain lengths while leaving faithful rollouts unchanged. • Artifacts. Probe weights, training configurations, evaluation harness, and rollout caches released across all four domains. 2 Related Work Chain-of-thought faithfulness. 20 flip final answers via biased prompts while leaving the chain unchanged; 14 introduce the forced-answering protocol we adopt for label generation (§ 4 ); 4 report large-scale faithfulness audits on production reasoning models. We refer to the post-commitment elaboration these studies measure as reasoning theater . This thread is diagnostic; ProFIL acts on the same signal to reduce it during training. Probes as reward signals. Probes recover latent knowledge [ 3 , 2 ] and steer behavior at inference [ 15 , 18 ] . Goodfire’s RLFR (Reinforcement Learning from Feature Rewards; 8 ) pushed multi-head attention probes from inference into RL training. Our probe architecture descends from RLFR; the key departure is that we probe the frozen base, not the policy being trained. Inference-time steering methods such as Contrastive Activation Addition (CAA; 18 ) and Inference-Time Intervention (ITI; 15 ) suppress behaviors at the forward-pass level but leave policy weights untouched, so the model regenerates the suppressed behavior on the next call. ProFIL eliminates theater durably by changing what the policy learns. Process reward models. Process Reward Models (PRMs; 21 , 16 ) score step correctness ; ProFIL scores step commitment . A step can be correct and performative simultaneously: a PRM rewards it, ProFIL filters the rollout containing it. The two are complementary. Length control as the closest auxiliary signal. Brevity and length-control penalties [ 12 , 1 ] target the same surface phenomenon (over-long chains) but operate on token count rather than commitment. Our LiveCodeBench ablation (§ 5.3 ) shows they worsen theater, isolating commitment-detection as the operative signal. 3 Background GRPO. GRPO [ 19 , 6 , 12 ] samples K rollouts per prompt, scores each with a verifier (a binary correctness checker, e.g., string match for GSM8K, test-case execution for LiveCodeBench), and updates the policy via group-relative advantage. Because the verifier cannot score non-verifiable behaviors like faithfulness, ProFIL gates which rollouts contribute to the update (§ 4 ). We define our two metrics, the performativity ratio (fraction of post-commitment steps) and the faithful fraction (fraction of chains with ratio \leq\delta ), formally in § 4 . 4 ProFIL: Probe-Filtered Reinforcement Learning Figure 2: ProFIL turns a frozen-base probe into a training signal. (1) Forced-answering yields verifier-derived step-level performativity labels (no human annotation): the first step whose forced answer is correct defines the commitment point ; later steps are performative. (2) A gated multi-head attention probe (RLFR-style; 8 ) is trained once on

Read the complete text