Learning in Low-Dimensional Subspaces:

Orthogonal Bottlenecks for Reinforcement Learning · 2026

Read Learning in Low-Dimensional Subspaces: by Orthogonal Bottlenecks for Reinforcement Learning (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26012.txt.

Introduction

Todorov and Sabatelli squeeze reinforcement-learning encoders through a fixed straight bottleneck. The bottleneck forces features into a low-dimensional slot, with no extra losses, pretraining or tweaks to the learner. Theory says that when the slot is wider than the true rank of the best value function, power survives and learning stays the same in low-dimensional form. Tests on single-task and multi-task trials match or beat baselines. The trick thus shapes what the learner sees, cheaply, without touching how it learns.

Catalog record

CREATOROrthogonal Bottlenecks for Reinforcement Learning
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58622
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26012

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning Abstract Deep reinforcement learning (RL) agents commonly rely on high-dimensional neural representations, despite growing evidence that task-relevant value and policy structure may be intrinsically low-dimensional. In this work, we present a simple yet effective representation-level prior that inserts a fixed orthonormal projection to constrain encoder features to a low-dimensional subspace, requiring no auxiliary objectives, pretraining, or changes to the underlying RL algorithm. Under a linear realizability assumption, we prove that when the bottleneck dimension exceeds the intrinsic rank of the optimal value function in feature space, the bottleneck preserves expressivity and leaves the induced gradient dynamics unchanged up to an equivalent low-dimensional parameterization. Empirically, we find that across both single and multi-task benchmarks, baseline performance is either matched or improved once the bottleneck dimension exceeds a small task-dependent threshold; in many cases, value representations can be compressed to extremely low dimensions without loss, and the minimal sufficient dimension depends far more on environment complexity than encoder width. In addition, we analyze representation geometry and find that orthogonal bottlenecks stabilize feature norms and are associated with higher effective rank. Together, these results support a representation-space interpretation of the manifold hypothesis in reinforcement learning and position orthogonal bottlenecks as a lightweight, architecture-agnostic mechanism for shaping RL representations. s z\in\mathbb{R}^{D} h=B^{\top}z\in\mathbb{R}^{k} \pi_{\psi}(a|s) V_{\xi}(s) \phi_{\theta} B^{\top} Policy Head \psi Value Head \xi Figure 1 : A simple, visual representation of the orthogonal bottleneck for deep reinforcement learning adapted to a typical Actor-Critic architecture. After encoding the state into features z\in\mathbb{R}^{D} , a fixed orthonormal projection constrains the representation to a k -dimensional subspace before feeding the policy and value heads. 1 Introduction Reinforcement learning (RL) agents are routinely equipped with highly over-parameterized neural representations, even when solving tasks whose underlying decision structure is comparatively simple. While deep networks offer the flexibility to model complex policies and value functions, it is natural to question to what extent standard deep RL architectures allocate representational capacity far beyond what the task itself demands. This mismatch between network capacity and task complexity aligns naturally with the manifold hypothesis in machine learning, which posits that high-dimensional data and learned representations often concentrate near low-dimensional manifolds embedded in ambient space ( 18 ; 7 ; 12 ; 36 ) . Recent evidence suggests that such low-dimensional structure also emerges in deep RL. On the policy side, 39 and 50 show that the space of behaviorally distinct policies realized by RL agents is effectively low-dimensional, allowing policy networks to be compressed by several orders of magnitude in parameter space without loss of behavioral expressivity. On the representation learning side, both model-free and model-based approaches aim to recover compact latent state manifolds that reflect task dynamics, typically through auxiliary or contrastive objectives that guide an encoder toward structured representations ( 42 ; 62 ; 10 ) . These approaches, however, generally treat the manifold as a structure to be either discovered through optimization or to be recovered post hoc via generative modeling. In this work, we take a different approach and study an alternative perspective: rather than encouraging the network to uncover a low-dimensional structure, we impose one explicitly through the architecture itself at the representation level. Among the many possible ways to impose a low-dimensional bottleneck on learned representations, our focus is on fixed orthonormal subspaces as they allow for representations that capture the intrinsic subspace of a task without redundancy or distortion. In fact, any linear map onto a k -dimensional subspace can be factorized into an orthonormal basis followed by a diagonal scaling; retaining only the orthonormal component yields a full-rank bottleneck in which all directions are treated uniformly and no single eigendirection dominates the representation. Orthogonal projections also enjoy favorable geometric properties, with random orthogonal maps approximately preserving distances with high probability, as formalized by the Johnson-Lindenstrauss lemma ( 26 ; 51 ; 16 ) , and orthogonal weight matrices are known to improve conditioning and gradient propagation in deep networks through non-expansiveness ( 47 ; 23 ) . Motivated by these observations, we study a simple architectural inductive bias, namely, after an encoder produces features z\in\mathbb{R}^{D} , we project them onto a fixed orthonormal basis B\in\mathbb{R}^{D\times k} with B^{\top}B=I_{k} and k\leq D via h=B^{\top}z , and feed only the compressed representation h\in\mathbb{R}^{k} to all downstream value and/or policy heads. This explicitly constrains the agent to operate within a k -dimensional orthogonal representation subspace, without altering the underlying learning algorithm or training objective. Figure 1 provides a visual overview of this architecture in a standard actor-critic setting. Our contributions are as follows. First, we provide theoretical guarantees showing that if the optimal value function is linearly realizable in feature space with intrinsic rank r , then a fixed orthogonal bottleneck of dimension k\geq r preserves expressivity and does not alter the induced gradient dynamics of the effective feature-to-representation mapping. Second, we empirically validate this approach across both single-task and multi-task benchmarks and multiple deep RL algorithms. We show that once k exceeds a small task-dependent threshold, low-dimensional orthogonal subspaces typically match (and sometimes improve upon) baseline performance, while yielding more stable and uniformly utilized representations as measured by diagnostics such as feature norms and effective rank. 2 Related Work Our work connects to two broad lines of research. The first studies low-dimensional structure in reinforcement learning, either in policy space or in learned state and value representations, motivated by the manifold hypothesis. The second investigates the role of orthogonality in deep networks as a mechanism for stabilizing signal and gradient propagation. While both bodies of work provide important insights into representation structure and training dynamics, they have largely been explored independently. Our approach sits at their intersection, using fixed orthonormal projections as a representation-level inductive bias in deep reinforcement learning. Low-dimensional structure and representations. A large body of work in reinforcement learning assumes that task-relevant variability is intrinsically low-dimensional and exploits this structure to design compact policy and state representations, often motivated by the manifold hypothesis ( 40 ; 12 ) . Unsupervised skill- and option-discovery methods explicitly construct low-dimensional latent spaces of behaviors or tasks using information-theoretic or variational objectives to encourage diversity and structure in the learned policy space ( 13 ; 21 ; 1 ; 11 ; 29 ) . More recent work makes a policy-manifold perspective explicit by learning generative models over policy parameters and compressing the policy space into low-dimensional latent codes ( 45 ; 39 ; 50 ) . Parallel lines of research in model-based and transfer reinforcement learning learn latent state, action, or policy embeddings to simplify control and enable fast adaptation across related tasks ( 63 ; 5 ; 46 ) . State (and state-action) representation learning methods similarly aim to recover compact, task-relevant features, typically through contrastive, predictive, or reconstruction-based objectives ( 42 ; 62 ; 10 ) . On the theory side, work on linearly realizable value functions and good feature representations formalizes a related notion of low-dimensional linear structure in feature space, primarily from a worst-case sample complexity perspective ( 9 ; 30 ; 55 ) . Finally, empirical studies of deep RL representations link geometric properties such as feature rank, isotropy, and neuron dormancy to loss of

Read the complete text