Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors

Viacheslav Sinii 1, Nikita Balagansky 1, Gleb Gerasimov 2,1, Daniil Laptev 1, Yaroslav Aksenov 1, Vadim Kurochkin 1, Alexey Gorbatovski 1, Boris Shaposhnikov 1, Daniil Gavrilov 1 · 2025

Read Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors by Viacheslav Sinii 1, Nikita Balagansky 1, Gleb Gerasimov 2,1, Daniil Laptev 1, Yaroslav Aksenov 1, Vadim Kurochkin 1, Alexey Gorbatovski 1, Boris Shaposhnikov 1, Daniil Gavrilov 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06608.txt.

Introduction

Tiny added vectors that steer a language model's reasoning are dissected by Viacheslav Sinii and eight co-authors in a 2025 machine learning paper. Reasoning training improves models, but how it rewires their internal computation is unclear. Steering vectors are small, learnable nudges inserted into the model's residual stream.

Trained with a reinforcement learning objective, these vectors recover much of the gain from full fine-tuning while staying easy to inspect. The last-layer vector behaves like a bias favoring certain first tokens, such as words that start a worked solution. The next layer's vector leaves attention patterns mostly alone and works through later components, boosting process words and structure symbols. The vectors even transfer to sibling models from the same family. The findings offer concrete evidence about what reasoning training changes inside a model.

Catalog record

CREATORViacheslav Sinii 1, Nikita Balagansky 1, Gleb Gerasimov 2,1, Daniil Laptev 1, Yaroslav Aksenov 1, Vadim Kurochkin 1, Alexey Gorbatovski 1, Boris Shaposhnikov 1, Daniil Gavrilov 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06608

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors Viacheslav Sinii 1, Nikita Balagansky 1, Gleb Gerasimov 2,1, Daniil Laptev 1, Yaroslav Aksenov 1, Vadim Kurochkin 1, Alexey Gorbatovski 1, Boris Shaposhnikov 1, Daniil Gavrilov 1 Abstract The mechanisms by which reasoning training reshapes LLMs’ internal computations remain unclear. We study lightweight steering vectors inserted into the base model’s residual stream and trained with a reinforcement-learning objective. These vectors match full fine-tuning performance while preserving the interpretability of small, additive interventions. Using logit-lens readouts and path-patching analyses on two models, we find that (i) the last-layer steering vector acts like a token-substitution bias concentrated on the first generated token, consistently boosting tokens such as “To” and “Step”; (ii) the penultimate-layer vector leaves attention patterns largely intact and instead operates through the MLP and unembedding, preferentially up-weighting process words and structure symbols; and (iii) middle layers de-emphasize non-English tokens. Next, we show that a SAE isolates features associated with correct generations. We also show that steering vectors (i) transfer to other models, (ii) combine across layers when trained in isolation, and (iii) concentrate magnitude on meaningful prompt segments under adaptive token-wise scaling. Taken together, these results deepen understanding of how trained steering vectors shape computation and should inform future work in activation engineering and the study of reasoning models. 1 1 footnotetext: 1 T-Tech, 2 HSE University. Corresponding author: [email protected] The code is available at https://github.com/corl-team/steering-reasoning . 1 Introduction Reasoning-oriented language models have recently made striking gains Jaech et al. 2024 ; Guo et al. 2025 . Many top systems are trained with reinforcement learning on verifiable tasks, especially mathematics, where correctness provides reliable rewards. Yet we still lack a mechanistic account of what this training changes inside the network. We train steering vectors – learned additive directions injected into the residual stream, while freezing the base model. This parameterization was shown to match the performance of fully-tuned models ( Sinii et al. 2025 ) and it isolates a small set of features that can be probed, ablated, and composed with mechanistic-interpretability tools. To isolate layer-wise effects, we fit one steering vector per layer \ell and measure its behavioral impact, then analyze in depth the layers with the clearest effects. Our findings are: • Layer-wise isolation of RL-induced gains. We insert a steering vector at a single layer and report the performance achievable by this minimal intervention, averaged across six modern math benchmarks. • Two mechanisms for single-layer steering. Single-layer steering operates via two distinct mechanisms: all pre-final layers downweight non-English tokens, while the final layer modifies the first generation token. • Last layer behaves like first-token substitution. The final-layer vector acts at unembedding, boosting opening tokens (e.g., “To”/“Step”); simply prefixing that token recovers \sim 10–11 points – about three-quarters of the explicit last-layer gain. • Penultimate-layer vector acts through the MLP. The effect is mediated almost entirely by the MLP, with minimal reliance on attention. • Properties of steering vectors. They compose across layers, transfer to other models, and, with adaptive magnitude, fire selectively on specific tokens. 2 Related Work Reinforcement learning with verifiable rewards. Jaech et al. 2024 demonstrated the striking performance of RL-tuned reasoning models, sparking a wave of follow-ups that develop these models ( Guo et al. 2025 ; Zeng et al. 2025 ; Liu et al. 2025a ; Hu et al. 2025 ) . Subsequent work has examined why this training is effective, analysing model behaviour and the sources of its gains ( Wang et al. 2025 ; Ye et al. 2025 ; Shao et al. 2025 ; Liu et al. 2025b ) . We contribute with a mechanistic study of the changes induced by reasoning training. Steering vectors are small additive perturbations to the residual stream that modulate model behavior. They are widely viewed as feature amplifiers – strengthening existing computations rather than introducing new mechanisms – and have been used to toggle or amplify reasoning-like behaviors ( Venhoff et al. 2025 ; Ward et al. 2025 ) . A common way to obtain them is contrastive extraction from activation pairs (e.g., positive vs. negative sentiment) ( Turner et al. 2023 ; Panickssery et al. 2023 ; Liu et al. 2023 ; Zou et al. 2023 ) . Beyond extraction, steering directions can also be trained : optimized with preference data for controllable generation ( Cao et al. 2024 ) , or learned as simple additive vectors that surface latent behaviors such as step-by-step reasoning or self-reflection ( Mack & Turner 2024 ; Engels et al. 2025 ; Betley et al. 2025 ) . In this work, we interpret steering vectors trained with GRPO-like objective using standard tools from mechanistic interpretability – logit-lens to read out token-level effects ( nostalgebraist 2020 ) , path patching to localize circuits ( Wang et al. 2022 ) , and circuit-style analyses in the QK/OV framework ( Elhage et al. 2021 ) . 3 Background Figure 1: Single-layer steering. Mean accuracy on six benchmarks for Qwen2.5-Math-7B when training a single vector s_{\ell} at layer \ell with all other layers frozen. Mid-layer vectors yield the largest gains but never match all-layer steering, indicating the improvement is distributed across layers. Recent work has shown that training lightweight steering vectors can match the performance of fully trained models ( Sinii et al. 2025 ) . Concretely, a vector s_{\ell}\in\mathbb{R}^{d} is added to the output residual stream of \ell ’th layer, and all other weights remain fixed. They used RLOO ( Ahmadian et al. 2024 ) objective to train reasoning models \nabla_{\theta}J=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_{\theta}(\cdot\mid x)}\bigl[\,a(x,y)\,\nabla_{\theta}\log\pi_{\theta}(y\mid x)\,\bigr], where the advantage is defined as a(x,y)=r(x,y)-b(x),\qquad b(x)=\frac{1}{N}\sum_{y}r(x,y). Here r(x,y) is the scalar reward for completion y on prompt x , and b(x) is the per-prompt baseline for variance reduction. Sinii et al. 2025 argue that this parameterization localizes training-induced changes in the model’s internal computations, making the intervention easier to interpret. We adopt this setup to learn per-layer steering vectors for our interpretability study. Diff-Diff CosSim Diff-Vector CosSim Figure 2: Steering Vector Persistence. For each steering layer i (color encodes i ; warm = early, cool = late) and each target layer \ell on the x -axis, we compute the mean cosine similarity of the per-token change in hidden representations \Delta F_{<\ell,i} . Left: similarity between \Delta F_{<l,i}(x) and the dataset mean \mathbb{E}_{x}[\Delta F_{<l,i}(x)] , showing how aligned the per-token shifts are. Right: similarity between \Delta F_{<l,i}(x) and the layer- \ell steering vector s_{\ell} , showing the alignment of the shifts with the layer’s own steering vector. 4 Single-layer steering vectors Setup. We study two base models – Qwen2.5-Math-7B ( Team 2024 ) and Llama3.1-8B-Instruct ( Grattafiori et al. 2024 ) . Models are trained on the DeepScaleR dataset ( Luo et al. 2025 ) with the sampling temperature \tau=1.0 , a 4K context window for Qwen2.5-Math-7B, and 8K for Llama3.1-8B-Instruct. Rewards are assigned with Math-Verify 1 1 1 https://github.com/huggingface/Math-Verify . We used 128 prompts and 16 generations per gradient step. Evaluation spans six math benchmarks: AIME24/25, AMC23, MATH500 ( Hendrycks et al. 2021 ) , MinervaMath ( Lewkowycz et al. 2022 ) , and OlympiadBench ( He et al. 2024 ) . We report the mean score across these benchmarks. For MATH500, MinervaMath,

Read the complete text