InT : Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning
Matthew Y. R. Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, Aviral Kumar · 2026
Read InT : Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning by Matthew Y. R. Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, Aviral Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14209.txt.
Introduction
Matthew Yang and co-authors improve how AI models learn from their own reasoning errors in 2026. Their method is called intervention training, or InT. It steps into a flawed reasoning trace at one step. That step is swapped for a correct one. The model then writes a new ending from that fixed step. These new endings can reach the right answer where the first try failed. The aim is better credit assignment for each step.
Catalog record
| CREATOR | Matthew Y. R. Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, Aviral Kumar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-14209
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
InT : Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning Matthew Y. R. Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, Aviral Kumar Abstract Figure 1 : Intervention training ( InT ) for improving credit assignment. InT proposes single-step interventions to replace incorrect intermediate steps in reasoning traces (1). Conditioned on these localized corrections, the model can generate counterfactual continuations that succeed where the original failed (2). We then perform supervised fine-tuning on these interventions, enabling effective credit assignment by upweighting the likelihood of the interventions in place of the mistakes. \absfont Abstract: Outcome-reward reinforcement learning (RL) has proven effective at improving the reasoning capabilities of large language models (LLMs). However, standard RL assigns credit only at the level of the final answer, penalizing entire reasoning traces when the outcome is incorrect and uniformly reinforcing all steps when it is correct. As a result, correct intermediate steps may be discouraged in failed traces, while spurious steps may be reinforced in successful ones. We refer to this failure mode as the problem of credit assignment . While a natural remedy is to train a process reward model, accurately optimizing such models to identify corrective reasoning steps remains challenging. We introduce Intervention Training ( InT ), a training paradigm in which the model performs fine-grained credit assignment on its own reasoning traces by proposing short, targeted corrections that steer trajectories toward higher reward. Using reference solutions commonly available in mathematical reasoning datasets and exploiting the fact that verifying a model-generated solution is easier than generating a correct one from scratch, the model identifies the first error in its reasoning and proposes a single-step intervention to redirect the trajectory toward the correct solution. We then apply supervised fine-tuning (SFT) to the on-policy rollout up to the point of error concatenated with the intervention, localizing error to the specific step that caused failure. We show that the resulting model serves as a far better initialization for RL training. After running InT and subsequent fine-tuning with RL, we improve accuracy by nearly 14% over a 4B-parameter base model on IMO-AnswerBench, outperforming larger open-source models such as gpt-oss-20b. 1 Introduction Post-training large language models (LLMs) with reinforcement learning (RL) has proven to be a highly effective strategy for improving their reasoning capabilities. As RL is scaled to more challenging tasks, training becomes increasingly dominated by incorrect rollouts, which are often long-horizon trajectories composed of many smaller, structured reasoning steps. Many such rollouts contain substantial portions of correct reasoning, yet outcome-based RL treats these correct steps the same as the critical mistakes that cause failure. This indiscriminate penalty can lead to undesirable behaviors including increased verbosity or frequent, premature shifts in the reasoning process, which have been widely observed when applying outcome-reward RL [ 5 , 49 ] . Moreover, on sufficiently difficult problems where no correct rollout is produced at all, outcome-reward RL provides no learning signals, as advantages collapse to zero. In principle, addressing this challenge requires solving the problem of credit assignment : if we could identify the intermediate step at which a reasoning trace goes astray and selectively penalize that step while reinforcing other possible promising steps, we could drive the model toward more favorable outcomes. However, existing approaches to credit assignment are difficult to apply in the LLM setting. For reasoning traces, estimating the “credit” (i.e., value) of each step typically requires running multiple branched rollouts conditioned on a given prefix, which is prohibitively expensive [ 18 , 28 , 21 ] . Prior work attempts to amortize this process by training explicit value functions (or process reward models) [ 39 , 28 ] , but how we should train such functions over long reasoning traces remains an open question. Moreover, even with a reasonably accurate value function, optimizing it to identify an alternate step is itself challenging [ 48 , 63 ] , because of the vast space of possible future steps. How can we perform credit assignment without training a value function? Rather than explicitly learning a value function, we ask the model itself to identify how failed trajectories can be locally corrected. This approach relies on an LLM’s ability to implicitly perform value estimation and optimization jointly, in order to propose an alternative, improved step, which we refer to as a corrective intervention . Our main idea is to reduce the task of generating end-to-end solutions to the substantially easier task of verifying individual steps, which can be accomplished via textual comparison against a reference solution (i.e., a textual “diff”). In our method, we instruct a base model to analyze the differences between a reference solution and an incorrect reasoning trace that it previously generated, with the goal of identifying the first point at which the trace goes wrong and proposing a single-step corrective intervention. When conditioned on this intervention, the model can then generate counterfactual reasoning traces that succeed from right before the point of failure even when the original rollout failed. We then train the model to internalize these interventions by applying supervised fine-tuning (SFT) on them, followed by RL. By fine-tuning the model on targeted single-step interventions, we selectively reduce the likelihood of the incorrect reasoning steps that the base model would otherwise produce, shifting it toward more favorable steps instead. Our approach remains notably simple and computationally efficient, as it avoids branched rollouts, explicit value-function training, or modifying the RL objective to include step-level rewards. Throughout this procedure, we never rely on a larger model; instead, we leverage the asymmetry [ 41 ] in task difficulty between instruction-following, verification, and generation within the same model to perform credit assignment. We call this approach Intervention Training ( InT ). We compare InT to several alternative approaches that leverage reference solutions for learning, including methods that train policies directly on reference solutions or on self-generated reflections conditioned on those solutions. In settings where the base model is already strong (e.g., Qwen3 for mathematics) and the SFT dataset is relatively small ( \approx 1K examples), we observe that InT makes particularly effective use of reference solutions to improve pass@ k performance, which sets up the fine-tuned model for online RL. Notably, this coincides with InT producing more “on-policy” trajectories (those with high likelihood under the base model), whereas other approaches tend to produce more “off-policy” ones. After online RL, InT improves performance by nearly 10% on average across four challenging mathematical reasoning benchmarks, including a \sim 14% gain on IMO-AnswerBench [ 30 ] , which consists of IMO-level problems curated by former medalists. These results demonstrate InT as a simple and effective paradigm for improving the reasoning capabilities of LLMs, through improved credit assignment. 2 Preliminaries, Notation, and Problem Statement Learning objective. Training LLMs for reasoning typically involves an LLM \pi , model-generated reasoning traces \mathbf{y}\sim\pi(\cdot|\mathbf{x}) produced on input prompts \mathbf{x}\sim\rho , and a binary reward r(\mathbf{x},\mathbf{y})\in\{0,1\} quantifying the correctness of the final answer. The goal is to maximize average reward, given by: \max_{\pi}\penalty\ \penalty\ J(\pi):=\mathbb{E}_{\mathbf{x}\sim\rho,\mathbf{y}\sim\pi(\cdot|\mathbf{x})}[r(\mathbf{x},\mathbf{y})]. We can maximize the objective above by updating the policy with the policy gradient [ 50 ] : \displaystyle\pi^{\prime}\leftarrow\pi+\alpha\cdot\mathbb{E}_{\mathbf{x}\sim D_{\text{train}},\mathbf{y}\sim\tilde{\pi}(\cdot|\mathbf{x})}[r(\mathbf{x},\mathbf{y})\cdot\nabla_{\pi}\log\pi(\mathbf{y}|\mathbf{x})], (1) where D_{\text{train}} is the set of training problems and \tilde{\pi} is the policy used to generate samples for training. In