Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning
Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Ling Pan · 2025
Read Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning by Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Ling Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24061.txt.
Introduction
Jiashun Liu and co-authors argue that dead neurons in learning agents should be counted by their gradients, not their outputs, in a 2025 paper. Deep reinforcement learning agents often lose neurons mid-training, and dead neurons cannot adapt when the world changes. The standard gauge watches activation levels, which works for plain networks but fades in fancier ones.
Their metric, GraMa, measures gradient magnitude per neuron, a direct read on learning capacity rather than mere firing. It exposes stubborn inactivity across residual networks, diffusion models, and varied activation functions. Resetting the neurons GraMa flags, a procedure the team calls ReGraMa, lifted learning across several algorithms and testbeds, from simulated robots to control suites. The shift is small to state and wide in reach: watch the will to learn, not the noise.
Catalog record
| CREATOR | Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Ling Pan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24061
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Ling Pan Abstract Deep reinforcement learning (RL) agents frequently suffer from neuronal activity loss, which impairs their ability to adapt to new data and learn continually. A common method to quantify and address this issue is the τ \tau -dormant neuron ratio, which uses activation statistics to measure the expressive ability of neurons. While effective for simple MLP-based agents, this approach loses statistical power in more complex architectures. To address this, we argue that in advanced RL agents, maintaining a neuron’s learning capacity , its ability to adapt via gradient updates, is more critical than preserving its expressive ability. Based on this insight, we shift the statistical objective from activations to gradients, and introduce GraMa ( Gra dient Ma gnitude Neural Activity Metric), a lightweight, architecture-agnostic metric for quantifying neuron-level learning capacity. We show that GraMa effectively reveals persistent neuron inactivity across diverse architectures, including residual networks, diffusion models, and agents with varied activation functions. Moreover, re setting neurons guided by GraMa ( ReGraMa ) consistently improves learning performance across multiple deep RL algorithms and benchmarks, such as MuJoCo and the DeepMind Control Suite. 1 Introduction Deep reinforcement learning (Deep RL) has achieved remarkable success across a variety of domains, including robotics ( Liu et al. 2021 ) , drone control ( Song et al. 2021 ) , and game playing ( Berner et al. 2019 , Schwarzer et al. 2023 ) . These advancements have been driven by the expressive power and adaptive learning ability of neural networks which effectively approximate and optimize value functions and/or policies ( Sokar et al. 2023 ) . However, recent studies have uncovered a critical and often underexplored challenge: as training progresses, subsets of neurons in these networks often experience a progressive loss of activity and become dormant ( Sokar et al. 2023 , Ma et al. 2024 , Qin et al. 2024 ) . This phenomenon reduces the learning capacity of the network, comprising its ability to adapt to non-stationary data distributions ( Nikishin et al. 2022 ) , which in turn hinders their ability to acquire new knowledge and adapt to evolving environments ( Abbas et al. 2023 ) . Despite its importance, quantifying and mitigating neuronal activity remains challenging due to its complex underlying mechanisms ( Lyle et al. 2023 , Nauman et al. 2024a , Lyle et al. 2024 , Obando Ceron et al. 2023 ) . To address this problem, a primary principle has been to restore a network’s learning ability by reactivating or resetting inactive neurons. These approaches span multiple granularities: model-level and layer-level resets that reinitialize specific layers, while straightforward, often lead to catastrophic forgetting ( Nikishin et al. 2023 ) . On the other hand, neuron-dependent resets (e.g., ReDo ( Sokar et al. 2023 ) ) target specific underperforming neurons ( Xu et al. 2023 ) and provide a finer-grained approach by selectively reinitializing a subset of neurons identified as dormant, which mitigates forgetting and maintains computational efficiency. The effectiveness of neuron-dependent resets hinges critically on having a reliable criterion to identify which neurons require initialization. Existing methods primarily rely on activation-based metrics, such as the \tau dormant neuron metric ( Sokar et al. 2023 ) , which measures neuronal inactivity based on activation values (i.e., the output of activation functions). It has demonstrated utility in standard architectures, and has been used to guide targeted neuron resets to restore activity in simple settings such as serial MLPs with ReLU activations ( Agarap 2018 ) , providing a simple means of maintaining learning capability without relying on auxiliary networks ( Nikishin et al. 2023 ) or models ( Lee et al. 2024 ) . By identifying neurons with weak or no activation and selectively reinitializing them, the activation-based dormancy metrics become a widely adopted tool for restoring learning capabilities and preventing performance plateau in standard deep RL settings ( Farias and Jozefiak 2025 , Liu et al. 2025 , Juliani and Ash 2024 ) . Figure 1: Dormant neuron metric struggles in advanced vision RL BRO-net agent ( Nauman et al. 2024b ) . Neuron resetting based on the dormant neuron index (ReDo) cannot restore the learning capacity of the agent, which limits its effectiveness. The curves record normalized score across 3 image-input tasks (15 runs per method), i.e., Dog Stand, Dog Walk, Dog Run. However, deep RL architectures have rapidly evolved beyond these simple architectures, integrating advanced components such as residual connections ( Nauman et al. 2024b , Lee et al. 2025a ) , mixture of experts (MoEs) ( Obando-Ceron et al. 2024 , Sokar et al. 2024 , Willi et al. ) , and diffusion-based models ( Ren et al. 2025 , Wang et al. 2024 ) to improve scalability and performance for tackling complex tasks involving continuous control and high-dimensional observations. Additionally, they also imperceptibly alter the statistical properties of neuron activations, creating a significant mismatch with traditional activity metrics. Through our extensive analysis, we find that resetting based on \tau -dormant neuron ratio (ReDo) reduces effectiveness in these newer architectures. As shown in Figure 1 , ReDo struggles to identify inactive neurons that have lost their learning capacity in the more advanced BRO-net architecture, which features residual paths and layer normalization, ultimately failing to achieve the intended enhancement. This mismatch arises because activation-based metrics focus solely on a neuron’s current output, evaluating its expressive capacity (how strongly it activates), while neglecting its learning capacity (how effectively it can adapt to new data distributions). Consequently, it becomes particularly problematic as deep RL architectures evolve beyond simple feedforward networks to incorporate advanced structural elements, diverse activation functions, and sophisticated normalization techniques. In these modern architectures, a neuron’s activation magnitude often fails to accurately reflect its true learning potential. To address this mismatch, we propose a fundamental shift in perspective: from evaluating neurons based on their outputs to evaluating their learning potential by leveraging gradient magnitude. While activation values only capture what a neuron currently expresses, gradients tend to measure a neuron’s capacity in response to the given situation that directly drives parameter updates. This makes it a general and natural proxy for neuronal health across architectural variations. Building upon this insight, we introduce GraMa ( Gra dient Ma gnitude based Neuronal Activity Metric), a robust and lightweight framework for quantifying neuronal activity via the gradient magnitudes. GraMa maintains validity across diverse architectural patterns, making it well-suited for modern deep RL agents. In addition, GraMa imposes negligible computational and memory overhead by utilizing information already present in the optimization pipeline, which is a critical consideration for resource-intensive RL training. Leveraging GraMa ’s efficiency, we develop a targeted neuron reset mechanism, ( ReGraMa ), that selectively reinitializes inactive neurons that have lost their learning capacity during training. This mechanism demonstrates robust efficiency across various architectures, including the SAC variant ( Nadimpalli et al. 2025 ) , the residual BRO-net ( Nauman et al. 2024b ) , and the diffusion-based policy DACER ( Wang et al. 2024 ) . Our contributions are summarized as follows: • We show that the widely-adopted activation-based neuronal health measurements