Preserving Plasticity in Continual Learning with Adaptive Linearity Injection
Seyed Roozbeh Razavi Rohani, Khashayar Khajavi, Wesley Chung, Mo Chen, Sharan Vaswani · 2025
Read Preserving Plasticity in Continual Learning with Adaptive Linearity Injection by Seyed Roozbeh Razavi Rohani, Khashayar Khajavi, Wesley Chung, Mo Chen, Sharan Vaswani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-09486.txt.
Introduction
Seyed Roozbeh Razavi Rohani, Khashayar Khajavi, Wesley Chung, Mo Chen, and Sharan Vaswani fight a quiet failure of lifelong learning: networks slowly lose the ability to learn at all. Plasticity fades as tasks stream in, and deep linear networks resist that fade. Their method, AdaLin, borrows the resistance by letting each neuron blend linearity into its own activation.
Every neuron carries a learnable dial and a gate driven by its gradient flow. When signal weakens, more linearity flows in to keep learning alive. No new hyperparameters and no task boundaries are needed. Across shuffled and relabeled image benchmarks, class-split tasks, a deep residual network, and off-policy reinforcement learning, AdaLin clearly helped. Ablations show the per-neuron adaptation is the crucial part. The authors also probe which internal metrics track plasticity loss.
Catalog record
| CREATOR | Seyed Roozbeh Razavi Rohani, Khashayar Khajavi, Wesley Chung, Mo Chen, Sharan Vaswani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-09486
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Preserving Plasticity in Continual Learning with Adaptive Linearity Injection Seyed Roozbeh Razavi Rohani, Khashayar Khajavi, Wesley Chung, Mo Chen, Sharan Vaswani Abstract Loss of plasticity in deep neural networks is the gradual reduction in a model’s capacity to incrementally learn and has been identified as a key obstacle to learning in non-stationary problem settings. Recent work has shown that deep linear networks tend to be resilient towards loss of plasticity. Motivated by this observation, we propose Ada ptive Lin earization ( AdaLin ), a general approach that dynamically adapts each neuron’s activation function to mitigate plasticity loss. Unlike prior methods that rely on regularization or periodic resets, AdaLin equips every neuron with a learnable parameter and a gating mechanism that injects linearity into the activation function based on its gradient flow. This adaptive modulation ensures sufficient gradient signal and sustains continual learning without introducing additional hyperparameters or requiring explicit task boundaries. When used with conventional activation functions like ReLU, Tanh, and GeLU, we demonstrate that AdaLin can significantly improve the performance on standard benchmarks, including Random Label and Permuted MNIST, Random Label and Shuffled CIFAR 10, and Class-Split CIFAR 100. Furthermore, its efficacy is shown in more complex scenarios, such as class-incremental learning on CIFAR-100 with a ResNet-18 backbone, and in mitigating plasticity loss in off-policy reinforcement learning agents. We perform a systematic set of ablations that show that neuron-level adaptation is crucial for good performance, and analyze a number of metrics in the network that might be correlated to loss of plasticity. * * footnotetext: These authors contributed equally to this work. 1 Introduction Loss of plasticity in neural networks refers to the gradual decline in a model’s ability to learn and adapt to new data over time, a phenomenon that is particularly exacerbated in non-stationary learning settings such as continual learning and reinforcement learning ( Nikishin et al., 2023 ; Lyle et al., 2023 ; Lyle et al., 2024 ; Dohare et al., 2024 ) . To address this challenge, recent approaches have mostly focused on regularizing the network during training ( Kumar et al., 2023 ; Lewandowski et al., 2023 ; Chung et al., 2025 ) or periodically reinitializing a subset of weights to restore learning capacity ( Ash & Adams, 2020 ; Dohare et al., 2024 ) . While these methods can alleviate plasticity loss in controlled settings, they often come with limitations. Regularization can overly constrain the model, reducing its overall capacity and performance, while reinitialization strategies often depend on explicit task boundaries and can disrupt learned representations, making them less suitable for real-world applications where tasks evolve continuously. Consequently, an alternative line of research has investigated how activation functions influence plasticity loss and highlighted their central role in sustaining a model’s learning ability over time ( Abbas et al., 2023 ; Lyle et al., 2024 ; Lewandowski et al., 2024 ) . A straightforward example of this phenomenon appears in Figure 1(a) , where the same MLP trained on the Random-label MNIST task 1 1 1 In this benchmark, 1200 random images are chosen from the MNIST dataset. For each task, every image is assigned a new random label. The experiment measures continual learning performance (in terms of average task accuracy) across a sequence of tasks, with each task representing one complete training cycle on the randomized labels. exhibits drastically different outcomes depending on its activation function. Specifically, standard activations such as ReLU, GeLU ( Hendrycks & Gimpel, 2016 ) , and Tanh fail to sustain the network’s learning capacity, experiencing significant degradation over time. By contrast, alternatives like CReLU ( Abbas et al., 2023 ) and PReLU ( He et al., 2015 ) better maintain plasticity across tasks. CReLU achieves this by concatenating the positive and negative responses, ensuring that a portion of each neuron’s output remains unsaturated. PReLU, on the other hand, introduces a learnable slope for negative inputs. Recent findings also suggest that deep linear networks, where the activation function is simply the identity, are more resilient to loss of plasticity, but lack the expressive power of nonlinear functions . ( Lewandowski et al., 2023 ; Lewandowski et al., 2024 ; Dohare et al., 2024 ) . This observation has inspired methods such as deep Fourier features ( Lewandowski et al., 2024 ) , which embed a deep linear model into the network by outputting both sine and cosine components 2 2 2 A detailed description of the deep Fourier method can be found in Appendix C . . (a) The impact of different activation functions on plasticity loss on the random-label MNIST task. Activation functions such as CReLU and PReLU help sustain the network’s learning capacity over multiple tasks, while others experience significant degradation. This highlights the crucial role of activation function choice in maintaining plasticity and long-term learning ability in neural networks. (b) AdaLin , our proposed method, adaptively injects linearity into the network and and can be used in conjunction with any given activation function. The figure shows the continual learning performance corresponding to using AdaLin with common activation functions such as Tanh, ReLU, GeLU. Comparing to Figure 1(a) , we observe that AdaLin can help sustain high task accuracy over successive tasks in the random-label MNIST scenario. Figure 1: (a) Baseline comparison of activation functions and their vulnerability to plasticity loss in the random-label MNIST task. (b) Improved performance when combining AdaLin with these activations. Deep Fourier features effectively embed linear components by mapping each preactivation to two outputs, ensuring that, at any point, one of the two responses of a neuron is approximately linear. However, this fixed ratio of linear to non-linear outputs may not be optimal across different tasks. Motivated by the need to adaptively adjust this balance, in this work, we propose Ada ptive Lin earization ( AdaLin ), a simple yet effective method for dynamically balancing linear and non-linear behavior in neural network activations. With AdaLin , each neuron learns a linearity injection parameter that determines the extent of the linear component added to its base activation. This adaptive mechanism allows neurons to modulate the proportion of linearity in their response based on how close their activation is to the saturation region. This balance between the stability of linear models and the expressive power of non-linear activations fosters continual adaptation without losing plasticity across tasks. Figure 2 depicts an illustration of how AdaLin functions. Furthermore, biological studies highlight the importance of dynamic adaptability in neural responses. Neuroscience research shows that, unlike fixed activation functions in conventional deep learning models, biological neurons dynamically adjust their response profiles in reaction to contextual factors and neuromodulatory influences ( Ferguson & Cardin, 2020 ; Cheadle et al., 2014 ; McGinley et al., 2015 ) . A more comprehensive review and related neuroscientific findings on this topic can be found in Appendix H . While numerous studies ( Apicella et al., 2021 ; He et al., 2015 ; Trottier et al., 2017 ; Qiu et al., 2018 ) have explored activation functions that can be trained during the learning process, the absence of these dynamic methods in continual learning may contribute to challenges like loss of plasticity. Figure 2: Illustration of AdaLin ’s adaptive activation mechanism. (Left) A