Manifold-Guided Attention Steering

Ian Li Kapilesh Guruprasad Raunak Sengupta [email protected] [email protected] [email protected] Ninad Satish Loris D’Antoni Rose Yu [email protected] [email protected] [email protected] University of California, San Diego · 2026

Read Manifold-Guided Attention Steering by Ian Li Kapilesh Guruprasad Raunak Sengupta [email protected] [email protected] [email protected] Ninad Satish Loris D’Antoni Rose Yu [email protected] [email protected] [email protected] University of California, San Diego (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21770.txt.

Introduction

Ian Li and co-authors propose Manifold-Guided Attention Steering, an inference-time fix for reasoning errors in language models. Existing steering uses fixed correction vectors no matter where the model sits, harming correct steps as freely as wrong ones. They rest on one finding: attention outputs drift from a low-dimensional correctness manifold at error onset, and the drift compounds. By learning a subspace from paired traces and correcting only when drift passes a threshold, MAGS beats plain and static-steering rivals on math, code, and molecular benchmarks.

Catalog record

CREATORIan Li Kapilesh Guruprasad Raunak Sengupta [email protected] [email protected] [email protected] Ninad Satish Loris D’Antoni Rose Yu [email protected] [email protected] [email protected] University of California, San Diego
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38530
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21770

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Manifold-Guided Attention Steering Ian Li Kapilesh Guruprasad Raunak Sengupta [email protected] [email protected] [email protected] Ninad Satish Loris D’Antoni Rose Yu [email protected] [email protected] [email protected] University of California, San Diego Abstract Large language models frequently produce errors in reasoning tasks despite possessing the underlying knowledge required for correct reasoning. One possible approach to improve reasoning consistency is through activation steering. However, existing activation steering approaches apply fixed, pre-computed correction vectors, ignoring where the model currently sits along its generation trajectory; the result is indiscriminate perturbation that disrupts already-correct steps as freely as erroneous ones. We propose Ma nifold- G uided Attention S teering (MAGS), a trajectory-aware inference-time intervention grounded in a geometric observation: the output activations of specific attention heads diverge from a low-dimensional correctness manifold at the point of error, and this deviation compounds through subsequent steps. For each identified attention head, we learn a low-dimensional subspace from contrastive pairs of correct and incorrect traces that capture the directions along which error behavior deviates from correct behavior. During inference, we monitor each head’s proximity to this manifold and apply a targeted projection correction when deviation exceeds a learned threshold, steering the attention output back toward the correct subspace before the error propagates. MAGS consistently outperforms both unsteered baselines and static steering approaches across benchmarks spanning mathematical reasoning (MATH-500, GSM8K), code generation (HumanEval, MBPP), and molecular generation (SMILES), suggesting that correctness manifolds are a general feature of LLM attention geometry. 1 Introduction Large language models (LLMs) frequently produce reasoning errors in multi-step reasoning tasks despite possessing the underlying capability to solve them. Under repeated sampling, the same model and prompt that produce an incorrect solution will often produce a correct one [ 3 , 20 , 4 ] , suggesting that the capability is present, but its reliable expression is not. Process-level annotations further confirm that most errors arise at intermediate reasoning steps rather than from a terminal absence of knowledge [ 11 , 17 ] . Since errors arise in the generation process rather than from missing knowledge, correcting them at inference time is a natural and practical target. Existing activation steering methods [ 16 , 24 , 14 , 18 ] apply a fixed correction vector to the residual stream at generation steps. These approaches are well-suited for persistent, global behaviors (tone, style, sentiment) but are structurally mismatched to reasoning. A reasoning trajectory may proceed correctly for many steps before committing a localized error at step t^{\ast} ; applying constant corrections may corrupt the correct intermediate steps while offering no guarantee of intercepting the error. We hypothesize that reasoning errors manifest as a drift in a low-dimensional subspace of individual attention heads’ output space: correct and incorrect trajectories occupy geometrically separable regions, and the transition from correct to incorrect behavior follows a structured, low-rank direction. We perform diagnostic experiments to confirm this hypothesis and in fact discover that correct and incorrect trajectories are highly separable by a low-dimensional subspace of attention-head activations. This is consistent with mechanistic interpretability findings that individual attention heads are functionally specialized [ 5 , 19 ] , and with the linear representation hypothesis [ 13 ] , which posits that semantically meaningful distinctions are encoded along low-dimensional linear directions. Therefore, one should steer only when their attention outputs have drifted into the error subspace. We propose Manifold-Guided Attention Steering (MAGS) : an adaptive intervention that dynamically steers the attention head outputs when reasoning errors are detected. MAGS outperforms static steering baselines across reasoning benchmarks and molecular generation on three model families, including Llama, Gemma, and GPT-OSS. In summary, our contributions are as follows: 1. We hypothesize that reasoning errors manifest as structured drift in a low-dimensional subspace of individual attention heads’ output space, and confirm this hypothesis with diagnostic experiments showing that correct and incorrect trajectories are highly separable (Section 3 ). 2. We propose Manifold-Guided Attention Steering (MAGS), an adaptive inference-time mechanism that monitors attention heads for reasoning drift and applies dynamic correction only when needed (Section 4 ). 3. Empirically, MAGS consistently outperforms static steering baselines on benchmarks across three model families, by up to 10.8\% while incurring negligible inference overhead (Section 5 ). 2 Related Work We discuss existing inference-time steering methods and geometric interpretability work. Existing steering methods apply fixed corrections without error-detection mechanisms; existing interpretability work establishes the geometric structure that we exploit and extend for adaptive intervention. To our knowledge, MAGS is the first method to combine per-step detection with geometry-aware, conditional correction at the attention-head level. Figure 1 : Comparison of static and Manifold-Guided Attention Steering (MAGS) on an example problem. Step-by-step reasoning traces for a static baseline and MAGS. Blue boxes denote correct reasoning steps; red boxes denote erroneous ones. 2.1 Activation Steering and Inference-time Intervention Activation steering methods modify internal representations at inference time without updating model parameters. Activation Addition [ 16 ] adds a fixed difference vector to the residual stream throughout generation. Contrastive Activation Addition (CAA; 14 ) improves reliability by averaging difference vectors across many contrastive prompt pairs. Representation Engineering (RepE; 24 ) extracts principal steering directions from contrastive activations via PCA. Angular Steering [ 18 ] replaces additive correction with a 2D rotation applied uniformly across all layers. Inference-Time Intervention (ITI; 9 ) shifts attention head outputs along a probing direction to improve truthfulness. CREST [ 23 ] identifies reasoning-relevant attention heads and applies fixed steering vectors to them, but does not adapt the correction to the model’s current trajectory state. All of these methods share a common limitation: they steer along a fixed direction regardless of the current activation state. MAGS addresses this by introducing a dynamic proximity trigger that fires only when a head drifts toward the error subspace and applying a step-dependent correction whose direction is determined by the current activation’s projection onto the error subspace rather than a fixed vector (as illustrated in Figure 1 ). 2.2 Geometric Structure of Transformer Representations A growing body of work establishes that transformer representations have rich geometric structure that can be leveraged for analysis and intervention. Mechanistic interpretability studies decompose transformer computation into interpretable circuits. 5 show that attention heads implement primitive operations (copying, retrieval, inhibition) whose outputs compose additively in the residual stream. 19 shows that multi-step tasks are implemented by sparse circuits across a small number of heads. Together, these results suggest that reasoning failures are likely attributable to specific heads in failure modes, which motivates our head-level intervention. The linear representation hypothesis [ 13 ] posits that semantically meaningful distinctions are encoded along low-dimensional linear directions in transformer representations. 2 show that truth has a linear representation findable by contrastive probing, establishing a precedent for our contrastive PCA construction. 24 confirm that high-level concepts, including reasoning quality, are linearly decodable from residual stream activations. MAGS extends this line of work to the per-head level, showing that the correct-to-error transition in reasoning trajectories is also linearly structured within individual head-output spaces. 3 Detecting Error Drift in Attention Heads We hypothesize that incorrect reasoning traces induce drift in the output of a subset of attention heads toward a low-dimensional error subspace, geometrically separable from the subspace occupied by correct traces. We empirically validate this hypothesis by constructing a contrastive error manifold per

Read the complete text