Stable Deep Reinforcement Learning via Isotropic Gaussian Representations
Ali Saheb, Johan Obando-Ceron, Aaron Courville, Pouya Bashivan, Pablo Samuel Castro · 2026
Read Stable Deep Reinforcement Learning via Isotropic Gaussian Representations by Ali Saheb, Johan Obando-Ceron, Aaron Courville, Pouya Bashivan, Pablo Samuel Castro (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-19373.txt.
Introduction
Deep reinforcement learning often wobbles because its targets and data keep shifting during training. Ali Saheb Pasand and four co-authors show that isotropic Gaussian embeddings, in which representations spread evenly in all directions, are provably helpful under such non-stationary targets. They support stable tracking of changing targets for linear readouts, maximize entropy for a fixed variance budget, and encourage balanced use of every dimension.
Building on that insight, the paper proposes Sketched Isotropic Gaussian Regularization, which shapes representations toward an isotropic Gaussian during training. Across varied domains, this simple and cheap method lifts performance under non-stationarity while cutting representation collapse, dormant neurons, and training instability. Readers get a lightweight stabilizer for agents facing shifting objectives.
Catalog record
| CREATOR | Ali Saheb, Johan Obando-Ceron, Aaron Courville, Pouya Bashivan, Pablo Samuel Castro |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-19373
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stable Deep Reinforcement Learning via Isotropic Gaussian Representations Ali Saheb, Johan Obando-Ceron, Aaron Courville, Pouya Bashivan, Pablo Samuel Castro Abstract Deep reinforcement learning systems often suffer from unstable training dynamics due to non-stationarity, where learning objectives and data distributions evolve over time. We show that under non-stationary targets, isotropic Gaussian embeddings are provably advantageous. In particular, they induce stable tracking of time-varying targets for linear readouts, achieve maximal entropy under a fixed variance budget, and encourage a balanced use of all representational dimensions–all of which enable agents to be more adaptive and stable. Building on this insight, we propose the use of Sketched Isotropic Gaussian Regularization for shaping representations toward an isotropic Gaussian distribution during training. We demonstrate empirically, over a variety of domains, that this simple and computationally inexpensive method improves performance under non-stationarity while reducing representation collapse, neuron dormancy, and training instability. Our code is available at IsoGaussian-DRL . \printArXivAffiliationsAndNotice * Equal contribution \dagger Equal advising 1 Introduction Deep reinforcement learning (deep RL) has achieved striking successes across a wide range of domains, including robotics, control, game playing, and large-scale decision-making ( 67 ; 6 ; 19 ; 59 ) . Despite this progress, optimization pathologies such as instability, plasticity loss ( 53 ) , neuron dormancy ( 61 ) , and representation collapse ( 52 ; 49 ) remain persistent and often limit both performance and scalability. A defining challenge of deep RL is its inherently non-stationary nature. Unlike supervised or standard self-supervised learning, the data distribution, learning targets, and optimization landscape evolve continually as the agent updates its policy and interacts with the environment ( 63 ; 65 ) . This non-stationarity has been repeatedly linked to degraded representation quality, unstable gradients, and brittle learning dynamics, ultimately reducing effective capacity over long training horizons ( 33 ; 54 ; 39 ; 8 ) . Most prior work addressing instability in deep RL focuses on algorithmic or architectural interventions, including modified update rules ( 28 ) , auxiliary objectives ( 9 ; 58 ) , target networks, or carefully designed architectural heuristics ( 14 ; 13 ; 35 ; 8 ) . While often effective, these approaches typically address downstream symptoms of instability and may introduce additional complexity or tuning requirements. In contrast, the role of representation geometry, specifically, how the statistical structure of learned representations interacts with non-stationary targets, has received comparatively limited attention in deep RL. Recent advances in self-supervised learning suggest that representation geometry plays a central role in stability and generalization. In particular, LeJEPA ( 4 ) demonstrates that enforcing simple statistical structure, namely isotropy and approximate Gaussianity, yields representations that generalize robustly across diverse downstream tasks. This setting closely mirrors deep RL, where the effective task and target distribution evolve over time and are not known a priori. Motivated by this parallel, we revisit instability in deep RL through the lens of representation geometry and ask: what properties should representations satisfy to remain stable under continuously evolving targets? We show that isotropic Gaussian representations are particularly well suited to this regime. For linear readouts tracking non-stationary targets, such representations minimize sensitivity to drift, maximize entropy under a fixed variance budget, and promote balanced utilization of representational dimensions. Together, these properties mitigate collapse, reduce neuron dormancy, and stabilize learning dynamics under distributional shift. These guarantees highlight isotropic Gaussian structure as a principled target for representations in non-stationary learning systems. To empirically study this hypothesis, we evaluate methods that encourage isotropic Gaussian structure in learned representations within standard deep RL pipelines. Our experiments span discrete and continuous control benchmarks, including the Arcade Learning Environment ( 7 ) and Isaac Gym ( 46 ) , and cover both value-based and policy-gradient methods such as Parallelized Q-Networks (PQN) ( 24 ) and Proximal Policy Optimization (PPO) ( 57 ) . Across settings, we observe consistent improvements in training stability, representation quality, and performance under non-stationarity. These results suggest that representation geometry, rather than algorithmic complexity or curvature estimation alone, plays a central role in stabilizing learning under non-stationarity such as deep RL. 2 Preliminaries Deep Reinforcement Learning. Deep RL addresses sequential decision-making problems in which an agent learns a policy mapping states to actions through interaction with an environment, typically using neural networks as function approximators ( 51 ) . This interaction is formalized as a Markov decision process (MDP) (\mathcal{S},\mathcal{A},P,r,\gamma) ( 56 ) . A policy \pi_{\theta}(a\mid s) is optimized via gradient-based methods, inducing a discounted state–action occupancy measure d_{\pi_{\theta}}(s,a) . As policy parameters are updated, both the policy and the induced data distribution evolve, resulting in a continuously shifting training distribution even under fixed environment dynamics. Most deep RL algorithms rely on bootstrapped learning targets. Value-based and actor–critic methods, including DQN ( 51 ) , PQN ( 24 ) , SAC ( 26 ) , and TD3 ( 23 ) , optimize objectives of the form \mathcal{L}_{\text{TD}}(\phi)=\mathbb{E}_{(s,a,r,s^{\prime})\sim d_{\pi_{\theta}}}\big[(Q_{\phi}(s,a)-y_{\theta,\phi}(s,a,r,s^{\prime}))^{2}\big]; where the target y_{\theta,\phi} depends on learned quantities. As a result, both the data distribution and learning targets evolve throughout training, introducing inherent non-stationarity and contributing to optimization instability ( 66 ; 27 ) . Empirically, this non-stationarity leads to representation drift, neuron dormancy, and loss of effective capacity, often manifested as rank collapse and degraded adaptability ( 42 ; 53 ; 52 ; 64 ) . Representation Learning in Deep RL. Deep RL agents typically decompose their networks into a shared representation h_{\eta}(s)\in\mathbb{R}^{d} and task-specific prediction heads, e.g., Q_{\phi}(s,a)=g_{\omega}(h_{\eta}(s),a) ( 21 ; 17 ) . Although objectives are defined over scalar signals such as returns or advantages, gradients propagate through the prediction head into the representation. Consequently, deep RL simultaneously performs control and representation learning under indirect, noisy, and non-stationary supervision ( 29 ) . Sampling s\sim d_{\pi_{\theta}} induces a distribution over representations z=h_{\eta}(s) , whose geometry evolves throughout training. 3 Isotropic Gaussian Representations Promote Stability Under Non-Stationarity We study the role of isotropic Gaussian representations by analyzing learning dynamics under drifting targets. We show that when embeddings are constrained to be isotropic and Gaussian, the resulting optimization dynamics has a stable fixed point, and the tracking error remains bounded and decreases over time. This provides a theoretical explanation for the empirical robustness of isotropic Gaussian representations under non-stationary training and motivates explicitly enforcing this type of representation in deep RL. Our analysis is based on the formal model in App. B , where representation learning is viewed as a tracking problem under drifting supervision. This analysis is inspired by Lemmas 1 and 2 in ( 4 ) , which show that, without knowledge of the downstream task, an isotropic representation is optimal for learning a wide range of downstream tasks. Building on this insight, we model non-stationarity as learning a sequence of tasks that evolve over time, and we seek conditions that ensure task changes induced by non-stationarity remain learnable. 3.1 Linear Tracking Under Non-Stationary Targets We consider a linear critic on top of a learned embedding, Q_{\theta}(s,a)=w^{\top}\phi(s,a) , where \phi(s,a)\in\mathbb{R}^{d} is the penultimate-layer embedding and w\in\mathbb{R}^{d} is the last-layer weight vector. The TD target is time-varying, y_{t}=r+\gamma Q_{\theta^{-}}(s^{\prime},a^{\prime}) , which induces non-stationarity in the learning problem. To characterize the resulting dynamics, define the second-order statistics \Sigma_{\phi}(t):=\mathbb{E}[\phi\phi^{\top}],\qquad b_{t}:=\mathbb{E}[\phi y_{t}] , and the expected critic