Invariant Reasoning Directions in Latent Trajectories of Language Models
Arun Vignesh Malarkkan, Manan Roy Choudhury, Utkarsh Byahut, Yash Ravindra Charde, Vivek Gupta, Yanjie Fu · 2026
Read Invariant Reasoning Directions in Latent Trajectories of Language Models by Arun Vignesh Malarkkan, Manan Roy Choudhury, Utkarsh Byahut, Yash Ravindra Charde, Vivek Gupta, Yanjie Fu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29164.txt.
Introduction
Malarkkan, Roy Choudhury, Byahut, Charde, Gupta and Fu investigate the hidden paths that latent models trace while thinking. Such models reason directly in hidden-state space over multiple steps, but the structure of those trajectories is poorly understood. Comparing stronger and weaker trajectories, the team find the useful signals concentrate in a low-rank structure, a few stable directions. Unconstrained updates wobble with paraphrases, checkpoints and perturbations.
Their framework, TILR, learns that low-rank stable subspace from contrastive trajectory differences. It restricts latent interventions to the subspace and suppresses poorly aligned updates through an adaptive gate, without retraining. Across six benchmarks, a small number of latent directions explained most of the variation between strong and weak trajectories. Intervening there made answers agree more under paraphrase, by about ten percent, and cut trajectory variance by up to half, while preserving accuracy.
Catalog record
| CREATOR | Arun Vignesh Malarkkan, Manan Roy Choudhury, Utkarsh Byahut, Yash Ravindra Charde, Vivek Gupta, Yanjie Fu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29164
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Invariant Reasoning Directions in Latent Trajectories of Language Models Arun Vignesh Malarkkan, Manan Roy Choudhury, Utkarsh Byahut, Yash Ravindra Charde, Vivek Gupta, Yanjie Fu Abstract Latent reasoning models perform multi-step inference directly in hidden-state space, yet the structure of these latent reasoning trajectories remains poorly understood. We show that contrastive refinement signals between stronger and weaker reasoning trajectories exhibit a highly concentrated low-rank structure, while unconstrained latent updates remain sensitive to paraphrases, checkpoint choice, and trajectory perturbations. These observations suggest that latent reasoning trajectories contain stable invariant directions mixed with unstable instance-specific variation. We introduce Trajectory-Invariant Latent Refinement (TILR) , a training-free intervention framework for identifying and manipulating stable reasoning directions in latent space. TILR first learns a low-rank invariant subspace from contrastive trajectory differences across inputs, then constrains latent interventions to this subspace while suppressing poorly aligned updates through an adaptive alignment gate. Across six reasoning benchmarks, we find that a small number of latent directions explain most variation between strong and weak reasoning trajectories. Interventions on these directions causally improve reasoning consistency and reduce trajectory instability under paraphrases and perturbations. TILR improves answer consistency under paraphrase by ∼ 10 % \sim 10\% and reduces latent trajectory variance by up to 50 % 50\% while preserving reasoning accuracy. These results support a geometric view of latent reasoning in which transferable reasoning behavior emerges from stable low-dimensional structure within hidden-state trajectories. † † affiliationnotice: Equal contribution 1 Introduction Latent reasoning models perform multi-step inference directly in hidden-state space, iteratively updating internal representations without generating explicit intermediate tokens. This paradigm enables reasoning without the token overhead of Chain-of-Thought prompting ( 22 ) , while retaining competitive performance across mathematical and multi-hop reasoning tasks. However, despite growing interest in latent reasoning ( 9 ) , little is known about the internal structure of the latent reasoning trajectories these models produce. In practice, semantically equivalent inputs often induce divergent latent trajectories, even when they produce the same final answer. Conversely, some latent directions remain stable across paraphrases, perturbations, and reference checkpoints. These observations suggest that latent reasoning trajectories may contain both stable reasoning-associated structure and unstable instance-specific variation. We hypothesize that reasoning behavior that remains stable across paraphrases, perturbations, and reference checkpoints is concentrated in a low-dimensional set of invariant latent directions, while unstable trajectory components correspond to paraphrase-sensitive, checkpoint-specific, or input-specific variation. Under this view, reasoning failures arise not because latent reasoning lacks useful signal, but because existing methods fail to distinguish stable reasoning-associated directions from unstable trajectory components. Recent latent refinement methods ( 21 ) steer trajectories using contrastive signals derived from stronger and weaker reference checkpoints. These methods improve reasoning performance without retraining, but the resulting latent updates are often unstable: semantically equivalent inputs can produce divergent trajectories, and refinement behavior can vary substantially across perturbations and reference choices. This suggests that existing methods operate on unconstrained latent directions that mix stable reasoning-associated structure with unstable variation. Figure 1 : Why invariant component? Our Perspective. We argue that latent reasoning trajectories contain a structured decomposition between stable reasoning-associated directions and unstable instance-specific components. Existing refinement methods treat contrastive updates as homogeneous vectors in the full embedding space, propagating both components equally. Empirically, we find that differences between strong and weak reasoning trajectories are highly concentrated in low-dimensional subspaces that remain stable across paraphrases and reference choices. This suggests that reasoning-quality information occupies a restricted set of latent directions distinct from generic trajectories. We introduce Trajectory-Invariant Latent Refinement (TILR) , a training-free intervention framework for identifying and manipulating stable reasoning-associated directions in latent space. TILR first identifies a low-rank invariant subspace in which stronger and weaker reasoning trajectories consistently differ across inputs. Contrastive updates are then constrained to this subspace, isolating stable contrastive structure from unstable trajectory variation. An adaptive alignment gate further modulates intervention strength according to how strongly the current trajectory aligns with the learned invariant structure, suppressing poorly aligned interventions while preserving stable latent directions. Our goal is not to identify semantic neurons or explicit reasoning circuits, but to study the geometric structure of latent reasoning trajectories through subspace-level interventions. To analyze latent reasoning beyond standard accuracy metrics, we introduce trajectory-level analyses that measure paraphrase consistency, latent trajectory variance, and sensitivity to reference checkpoint choice. Across six benchmarks, we find that a small number of latent directions explain most variation between strong and weak reasoning trajectories. Subspace interventions consistently affect reasoning consistency and trajectory stability under paraphrases and perturbations while preserving accuracy. Contributions. • We provide evidence that latent reasoning trajectories exhibit highly concentrated low-rank structure, with stable reasoning-quality differences occupying a restricted set of latent directions. • We introduce TILR, a training-free intervention framework that isolates and manipulates invariant reasoning-associated directions through subspace-constrained latent interventions. • We show through subspace interventions that these invariant directions play a functional role in reasoning consistency and trajectory stability. • We introduce trajectory-level analyses for studying latent reasoning invariance, including paraphrase consistency, trajectory variance, and checkpoint-pair sensitivity. 2 Related Work Latent reasoning. Chain-of-Thought prompting ( 22 ) proved the value of intermediate reasoning steps for LLMs, but each step consumes output tokens. Coconut ( 9 ) reformulates reasoning as a recursive update in embedding space, eliminating the per-step token cost while matching CoT accuracy. This includes methods that learn to generate discrete reasoning paths in latent space ( 6 ) and methods to internalize reasoning within hidden states during training ( 5 ) . TILR operates on the latent reasoning paradigm and is agnostic to how the backbone was trained; it requires only that the backbone produce a sequence of latent states that can be intervened on at inference time. Post-training inference-time refinement. Recent work modifies LLM behavior at inference time without parameter updates. Contrastive decoding ( 11 ) and its derivatives steer token-level generation by combining logits from stronger and weaker model variants. Activation steering ( 20 ; 17 ) edits intermediate representations along directions identified from contrastive examples, for alignment rather than reasoning quality. The recent work on post-refinement ( 21 ) adapts the contrastive principle to latent reasoning, using the difference between stronger and weaker checkpoints to steer Coconut’s continuous thoughts. AdaAnchor ( 19 ) introduces adaptive halting for latent reasoning, terminating refinement when anchor representations stabilize. Where prior methods accept the contrastive signal as given, TILR decomposes it into stable and unstable components and applies only the stable part. Subspace projection and invariance. Identifying low-rank structure in neural representations in domain adaptation ( 3 ) and continual learning ( 18 ; 12 ) are well-established, where projecting gradient updates onto subspaces preserves prior knowledge or filters distribution-specific variation. Invariant Risk Minimization ( 2 ) and the invariant prediction framework ( 15 ) formalize the principle that features whose predictive relationship is stable across environments generalize better than features merely correlated with the target. TILR draws on this principle without inheriting the optimization overhead: because the contrastive difference is directly observed via forward passes through reference models, the invariant directions can be extracted in closed form from a single SVD rather than learned through adversarial or bilevel optimization. 3 Background and Problem Setup Latent Reasoning - Coconut backbone. Latent reasoning models perform multi-step inference by iteratively updating a