Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Exploration
Chengqi Zheng, Haiyan Yin, Jianda Chen, Terence Ng · 2025
Read Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Exploration by Chengqi Zheng, Haiyan Yin, Jianda Chen, Terence Ng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05246.txt.
Introduction
Zheng and co-authors present a method for teaching AI agents new tasks in sequence without erasing old skills, in this 2025 paper. Agents must balance fast learning with stable memory. Their method, called SSDE, splits the network into frozen parts that carry old knowledge and free parts for new tasks.
A sparse coding scheme shares out the free capacity so each task gets enough room. The method also wakes up dormant neurons, units that have gone quiet, to restore flexibility as frozen parts pile up. The authors argue this mix keeps agents adaptable over long learning runs. Tests in the paper weigh the approach on continual learning tasks.
Catalog record
| CREATOR | Chengqi Zheng, Haiyan Yin, Jianda Chen, Terence Ng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05246
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Exploration Thanks: Chengqi Zheng, Haiyan Yin, Ivor Tsang, and Yew-Soon Ong are with the CFAR and IHPC, Agency for Science, Technology and Research (A*STAR), Singapore. Ivor Tsang and Yew-Soon Ong are also jointly affiliated with College of Computing and Data Science, Nanyang Technological University (NTU), Singapore. Thanks: Jianda Chen and Terence Ng are with the College of Computing and Data Science, Nanyang Technological University (NTU), Singapore Thanks: Haiyan Yin is the corresponding author (E-mail: [email protected]). Chengqi Zheng, Haiyan Yin, Jianda Chen, Terence Ng Abstract Continual Reinforcement Learning (CRL) is essential for developing agents that can learn, adapt, and accumulate knowledge over time. However, a fundamental challenge persists as agents must strike a delicate balance between plasticity , which enables rapid skill acquisition, and stability , which ensures long-term knowledge retention while preventing catastrophic forgetting. In this paper, we introduce SSDE , a novel structure-based approach that enhances plasticity through a fine-grained allocation strategy with S tructured S parsity and D ormant-guided E xploration. SSDE decomposes the parameter space into forward-transfer (frozen) parameters and task-specific (trainable) parameters. Crucially, these parameters are allocated by an efficient co-allocation scheme under sparse coding, ensuring sufficient trainable capacity for new tasks while promoting efficient forward transfer through frozen parameters. However, structure-based methods often suffer from rigidity due to the accumulation of non-trainable parameters, limiting exploration and adaptability. To address this, we further introduce a sensitivity-guided neuron reactivation mechanism that systematically identifies and resets dormant neurons, which exhibit minimal influence in the sparse policy network during inference. This approach effectively enhance exploration while preserving structural efficiency. Extensive experiments on the CW10-v1 Continual World benchmark demonstrate that SSDE achieves state-of-the-art performance, reaching a success rate of 95% , surpassing prior methods significantly in both plasticity and stability trade-offs (code is available at: https://github.com/chengqiArchy/SSDE ). Fig. 1 : Plasticity-stability trade-off on CW10-v1 benchmark : Stability refers to how well an agent retains learned knowledge, measured by task performance; plasticity quantifies how quickly an agent adapts to new tasks, measured by the normalized steps required to learn a task (Eq. 10 ). Our proposed method SSDE achieves state-of-the-art stability of 95%, demonstrating strong capability in mitigating forgetting. For plasticity , SSDE remains competitive with strong Behavior Cloning baselines, despite the latter having access to more data during experience replay. I Introduction From lifelong robotics to autonomous decision-making, real-world AI systems must continuously learn without forgetting past knowledge. While humans exhibit remarkable abilities to transfer knowledge across tasks and retain learned skills, AI models, particularly reinforcement learning (RL) agents, struggle in non-stationary environments, where past knowledge can be easily overwritten or lost [ 1 , 2 , 3 ] . Continual RL aims to address this challenge by enabling agents to learn sequentially without catastrophic forgetting [ 4 , 5 , 6 , 7 , 8 ] . However, achieving this requires striking a delicate balance between plasticity and stability in learning systems. Plasticity allows agents to quickly adapt to new tasks, while stability ensures that previously learned skills are retained [ 9 , 10 ] . Existing approaches to continual RL have pursued three main approaches: (i) rehearsal -based, which store and replay past experiences to reinforce prior knowledge [ 11 , 12 ] ; (ii) regularization -based, which impose constraints to prevent drastic updates that could overwrite previous learning; (iii) structure -based, which allocate dedicated sub-networks to different tasks within a shared parameter space [ 5 , 13 ] . While rehearsal and regularization-based methods provide some degree of stability, they offer limited control over parameter interference, as experience replay and constrained optimization can introduce unintended disruptions to learned policies, making long-term retention unreliable. In contrast, structure-based methods explicitly define task boundaries and allocate independent sub-networks, effectively minimizing interference and mitigating catastrophic forgetting. Moreover, structure-based approaches eliminate the need for storing or replaying past task data, making them particularly suitable for scenarios with strict memory constraints or data inaccessibility. One key property of structure-based methods is that they allocate sparse sub-networks for different tasks within a shared parameter space, enabling efficient knowledge retention while reducing interference [ 14 , 13 ] . PackNet [ 15 ] tackles this by iteratively pruning parameters after each task, retaining only the most crucial ones. CoTASP [ 16 ] employs task-driven sparse binary masks to dynamically regulate layer outputs, ensuring task-specific adaptation. These masks, initialized via sparse coding and dictionary learning, are continuously refined through gradients derived from RL objectives, optimizing network specialization over time. However, treating sub-network allocation as a unified process introduces a key limitation: as more tasks are added, the amount of trainable parameters progressively shrinks . This gradual capacity reduction restricts the model’s ability to adapt, ultimately compromising plasticity, especially in complex tasks where greater capacity is essential, as suggested by scaling laws [ 17 ] . Additionally, these methods often incur significant computational overhead, as sub-network allocation relies on costly procedures like pruning and gradient-based optimization. To enhance the plasticity of structure-based methods, it is crucial to not only allocate sufficient trainable parameters for new tasks but also effectively leverage previously trained parameters for inference. Achieving this balance is essential for maintaining the expressiveness of sparse sub-network policies and ensuring the long-term adaptability of continual RL models. In this paper, we present SSDE , a novel method for enhancing plasticity through “ S tructured S parsity and D ormant neuron-guided E xploration ”, designed to optimize the three core aspects of continual RL policies: (i) Allocation: SSDE introduces a fine-grained co-allocation strategy based on sparse coding, explicitly partitioning sub-network parameters into forward-transfer (fixed) and task-specific (trainable) components. This ensures sufficient capacity for acquiring new skills while maximizing knowledge transfer efficiency. (ii) Inference: SSDE incorporates a dedicated inference function with a novel trade-off parameter that dynamically balances forward-transfer and task-specific parameters. This prevents frozen parameters from overshadowing trainable ones, thereby expanding the solution space and enabling more flexible and adaptive inference strategies. (iii) Training: SSDE introduces a sensitivity-guided dormant neuron reset algorithm to enhance the expressiveness of sparse policies, which are inherently constrained in trainable capacity. By identifying and reactivating neurons that remain unresponsive to input sensitivity, SSDE mitigates the limitations of sparse sub-networks and improves their adaptability. Together, the strategic combination of fine-grained co-allocation and exploration with dormant neurons establish a robust foundation for SSDE to significantly enhance the plasticity - stability trade-off. We show SSDE not only achieves SOTA stability but also achieves competitive plasticity even when compared to strong behavior cloning baselines that benefit from data replay (Figure 1 ). We also show the consistency of SSDE’s performance across both v1 & v2 of Continual World benchmark (Table I & Table III ). A case study on co-allocated masks with structured sparsity highlights that our approach significantly improves parameter utilization while drastically reducing allocation time (Table II & Figure 4 ). Visualizations of sub-network masks further demonstrate that structural sparsity effectively captures task similarities (Figure 5 ). Finally, a comprehensive ablation study (Table IV ) confirms that SSDE’s core components are crucial for driving the success of the model. II Related Works Continual RL, a.k.a. lifelong RL, seeks to develop agents capable of