LASER: Low-Rank Activation SVD for Efficient Recursion
Ege Çakar, Ketan Raghu & Lia Zheng · 2026
Read LASER: Low-Rank Activation SVD for Efficient Recursion by Ege Çakar, Ketan Raghu & Lia Zheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17224.txt.
Introduction
Çakar, Raghu, and Zheng study the activation shape of tiny recursive models during unrolling. They find signals occupy a narrow subspace whose directions shift and can be tracked with power iterations. Weight-sharing focuses computation along a few dominant eigendirections, with focus varying sharply across sites. LASER maintains a low-rank basis via matrix-free tracking with a fidelity reset, cutting activation memory by about 60 percent with no real accuracy loss. The work raises questions about how recursive models allocate capacity during implicit reasoning.
Catalog record
| CREATOR | Ege Çakar, Ketan Raghu & Lia Zheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38497 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17224
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LASER: Low-Rank Activation SVD for Efficient Recursion Ege Çakar, Ketan Raghu & Lia Zheng Abstract Recursive architectures such as Tiny Recursive Models (TRMs) perform implicit reasoning through iterative latent computation, yet the geometric structure of these reasoning trajectories remains poorly understood. We investigate the activation manifold of TRMs during recursive unrolling and find that activations occupy an effectively linear, low-dimensional subspace whose principal directions can be tracked dynamically with cheap power iterations. This suggests that weight-sharing concentrates iterative computation along a small number of dominant eigendirections, and we find that this concentration varies sharply across computational sites. We exploit this structure through LASER (Low-Rank Activation SVD for Efficient Recursion), a dynamic compression framework that maintains an evolving low-rank basis via matrix-free subspace tracking with a fidelity-triggered reset mechanism, achieving ∼ 60 % {\sim}60\% activation memory savings with no statistically significant accuracy degradation. Our analysis raises questions about how recursive architectures allocate representational capacity during implicit reasoning, and whether this concentration can be exploited to improve the efficiency and stability of latent computation. Standard TRM X_{1}\in\mathbb{R}^{B\times D} X_{2}\in\mathbb{R}^{B\times D} \vdots X_{n}\in\mathbb{R}^{B\times D} Memory: O(nBD) LASER Compressed TRM Z_{1}\in\mathbb{R}^{B\times k} Z_{2}\in\mathbb{R}^{B\times k} \vdots Z_{n}\in\mathbb{R}^{B\times k} Q\in\mathbb{R}^{D\times k} k = retained rank, k\ll D Memory: O(nBk+Dk) Key result 60% less activation memory No significant accuracy loss Figure 1: LASER overview. Standard TRM training stores full activations X_{1},\dots,X_{n} , with each X_{i}\in\mathbb{R}^{B\times D} , across recursive steps. LASER instead stores compressed coefficients Z_{1},\dots,Z_{n} , where Z_{i}=X_{i}Q\in\mathbb{R}^{B\times k} , together with a shared low-rank basis Q\in\mathbb{R}^{D\times k} , where k\ll D is the retained rank. 1 Introduction Implicit reasoning, the ability to perform multi-step computation through iterative latent processing rather than explicit chain-of-thought generation, has emerged as a promising paradigm for both parameter-efficient and capable reasoning models. Tiny Recursive Models (TRMs) ( 5 ) exemplify this approach: a single transformer block is unrolled over many recursive time-steps, enabling deep computation with minimal parameters. While TRMs achieve strong performance on structural reasoning tasks, their training requires storing activations across all unrolled steps for backpropagation through time, creating a memory bottleneck that limits practical reasoning depth. This bottleneck raises a natural question: what is the intrinsic dimensionality of these stored reasoning trajectories? If activations across 24 recursive steps span a high-dimensional, nonlinearly curved manifold, then reducing memory requires powerful but expensive methods. However, if the manifold is low-rank and approximately linear, then the iterative reasoning process is far more constrained than the architecture nominally permits, and the memory problem becomes tractable. We find the latter. TRM activations occupy an effectively linear, low-dimensional subspace whose principal directions can be tracked dynamically with cheap power iterations. This finding has both interpretive and practical significance. Interpretively, it suggests that weight-sharing concentrates iterative computation along a small number of dominant eigendirections, raising questions about the effective utilization of reasoning depth. Practically, it motivates LASER (Low-Rank Activation SVD for Efficient Recursion), a dynamic compression framework that exploits this structure to achieve {\sim}60\% activation memory savings with negligible accuracy loss. This structure also suggests that recursive models are especially well-suited to activation compression. In standard backpropagation through time, activation memory grows directly with the number of recursive steps because a full hidden state must be retained at each step. LASER instead stores a compressed representation at each step together with a shared low-rank basis, so its advantage becomes larger as recursion depth increases. We evaluate this effect in the 24-step TRM setting used in prior work, both to remain comparable to the original architecture and to stay within our training budget. Our contributions are: • Low-rank geometry: TRM activations exhibit strong spectral decay and are well-approximated by a linear low-rank subspace. • Adaptive tracking: LASER tracks this evolving subspace with matrix-free power iteration and fidelity-triggered fallback. • Practical compression: LASER reduces activation memory by about 60% with no statistically significant loss in performance. • Quantization compatibility: TRM activations remain stable under INT8 quantization after low-rank projection. 2 Related Work 2.1 Activation Checkpointing and Quantization Standard memory-reduction techniques generally fall into two categories: Rematerialization and Quantization . Gradient Checkpointing (or Rematerialization) reduces memory by discarding intermediate activations and recomputing them during the backward pass. While effective, it imposes significant computational overhead when done aggressively, and we propose that LASER can be applied in parallel with non-aggressive checkpointing. Scalar quantization methods like ActNN ( 2 ) and GIST ( 4 ) compress activations by reducing precision (e.g., to 2-bit or 4-bit integers). While efficient, pure quantization treats activations as statistically independent scalars, ignoring the strong correlations between features. In this work, we demonstrate that quantization can complement subspace methods: LASER applies quantization to its low-rank residuals , yielding higher compression rates than either method alone. 2.2 Low-Rank Activation Compression A more structurally aware approach to compression exploits the low-rank nature of neural activations. If the activation matrix X\in\mathbb{R}^{N\times d} (where N=B\times L ) resides in a subspace of rank k\ll d , it can be stored as two smaller factors U\in\mathbb{R}^{N\times k} and V\in\mathbb{R}^{d\times k} . 2.2.1 Static and Randomized Projections Recent work has attempted to apply this principle to training. LANCE ( 1 ) employs a “one-shot” Higher-Order SVD (HOSVD) to fix a compression basis at initialization. This eliminates decomposition overhead, but assumes the activation subspace at epoch 0 is the same as at epoch 100. Our experiments show this stationarity assumption fails for recursive models, where the manifold evolves as the model learns to reason. GALE ( 6 ) uses randomized projections (sketching) to compress states. However, applying randomized sketching to activations at every layer introduces significant computational overhead. 2.2.2 Subspace Tracking Our method draws inspiration from signal processing techniques for subspace tracking, such as the Projection Approximation Subspace Tracking (PAST) algorithm ( 8 ) . Unlike LoRAct ( 7 ) , which uses sampling-based orthogonal decomposition, LASER applies a power-iteration scheme directly to the activation stream. Furthermore, we address the primary weakness of iterative tracking, error accumulation, by implementing an adaptive fallback: when the tracked subspace fails to explain the variance, we trigger a sparse, exact SVD update to realign the basis. While the above methods focus on compression as an engineering goal, our work additionally treats compressibility as a diagnostic : the degree to which activations admit low-rank approximation reveals structural properties of the implicit reasoning process itself. 3 Approach 3.1 The Geometry of Implicit Reasoning Trajectories We perform principal component analysis on activations collected from trained TRMs across all recursive time-steps. The spectral decay is rapid: at the MLP intermediate site (dimension 3072), the top 128 components capture the vast majority of activation variance (Figure 4 in the Appendix). This implies that out of 3072 available dimensions, the model’s reasoning trajectory is effectively confined to a subspace roughly 24 \times smaller. To rule out the possibility that a nonlinear manifold merely appears linear under PCA, we compared linear projection against nonlinear autoencoder baselines. The autoencoders provided no meaningful improvement in reconstruction quality, confirming that the low-rank structure is genuinely linear rather than a curved manifold that happens to be locally flat. This linearity is not coincidental—it arises from architectural induction biases. Smooth activations (GELU, SiLU) are locally linear over the operating range; LayerNorm constrains activations to a bounded manifold; and