Recovering Plasticity of Neural Networks via Soft Weight Rescaling

Seungwon Oh, Sangyeon Park, Isaac Han, Kyung-Joong Kim · 2025

Read Recovering Plasticity of Neural Networks via Soft Weight Rescaling by Seungwon Oh, Sangyeon Park, Isaac Han, Kyung-Joong Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04683.txt.

Introduction

Seungwon Oh, Sangyeon Park, Isaac Han, and Kyung-Joong Kim treat a quiet failure mode of neural networks in this two thousand twenty-five paper. As training proceeds, networks gradually lose plasticity, their capacity to learn new information. One main cause is unbounded growth of the weights, which also harms generalization and destabilizes optimization. Reinitializing the network restores plasticity but throws away everything learned. Their method, Soft Weight Rescaling, instead scales the weights down slightly at every learning step.

The authors prove theoretically that the rescaling bounds weight magnitudes and balances them across layers. Experiments on standard image classification benchmarks covered warm-start learning, continual learning, and ordinary single-task learning. In each setup the method improved performance over baselines suffering plasticity loss. The appeal is simplicity: no architecture change, no extra data, just a gentle, continuous brake on weight growth that keeps the network able to learn.

Catalog record

CREATORSeungwon Oh, Sangyeon Park, Isaac Han, Kyung-Joong Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04683

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Recovering Plasticity of Neural Networks via Soft Weight Rescaling Seungwon Oh, Sangyeon Park, Isaac Han, Kyung-Joong Kim Abstract Recent studies have shown that as training progresses, neural networks gradually lose their capacity to learn new information, a phenomenon known as plasticity loss. An unbounded weight growth is one of the main causes of plasticity loss. Furthermore, it harms generalization capability and disrupts optimization dynamics. Re-initializing the network can be a solution, but it results in the loss of learned information, leading to performance drops. In this paper, we propose Soft Weight Rescaling (SWR), a novel approach that prevents unbounded weight growth without losing information. SWR recovers the plasticity of the network by simply scaling down the weight at each step of the learning process. We theoretically prove that SWR bounds weight magnitude and balances weight magnitude between layers. Our experiment shows that SWR improves performance on warm-start learning, continual learning, and single-task learning setups on standard image classification benchmarks. 1 Introduction Recent works have revealed that a neural network loses its ability to learn new data as training progresses, a phenomenon known as plasticity loss. A pre-trained neural network shows inferior performance compared to a newly initialized model when trained on the same data ( Ash & Adams 2020 ; Berariu et al. 2021 ) . Lyle et al. 2024b demonstrated that unbounded weight growth is one of the main causes of plasticity loss and suggested weight decay and layer normalization as solutions. Several recent studies on plasticity loss have proposed weight regularization methods to address this issue ( Kumar et al. 2023 ; Lewandowski et al. 2023 ; Elsayed et al. 2024 ) . Unbounded weight growth is a consistent problem in the field of deep learning; it is problematic not only for plasticity loss but also undermines the generalization ability of neural networks ( Golowich et al. 2018 ; Zhang et al. 2021 ) and their robustness to distribution shifts. Increasing model sensitivity, where a small change in the model input leads to a large change in the model output, is also closely related to the magnitude of the weights. Therefore, weight regularization methods are widely used in various areas of deep learning and have been consistently studied. Weight regularization methods have been proposed in various forms, including additional loss terms ( Krogh & Hertz 1991 ; Kumar et al. 2023 ) and re-initialization strategies ( Ash & Adams 2020 ; Li et al. 2020b ; Taha et al. 2021 ) . The former approach adds an extra loss term to the objective function, which regularizes the weights of the model. These approaches are used not only to penalize large weights but also for other purposes, such as knowledge distillation ( Shen et al. 2024 ) . However, they can cause optimization difficulties or conflict with the main learning objective, making it harder for the model to converge effectively ( Ghiasi et al. 2024 ) . Liu et al. 2021 also proved that the norm penalty of a family of weight regularizations weakens as the network depth increases. Moreover, such methods require additional gradient computations, resulting in slower training. In addition, several studies argued that regularization methods could be problematic with normalization layers. For instance, weight decay destabilizes optimization in weight normalization ( Li et al. 2020a ) , and interferes learning with batch normalization ( Lyle et al. 2024b ) , both of which can hinder convergence. On the other hand, re-initialization methods are aimed at resetting certain parameters of the model during training to escape poor local minima and encourage better exploration of the loss landscape. Zaidi et al. 2023 demonstrated that re-initialization methods improve generalization even with modern training protocols. While re-initialization methods improve generalization ability, they raise the problem of losing knowledge from previously learned data ( Zaidi et al. 2023 ; Ramkumar et al. 2023 ; Lee et al. 2024 ; Shin et al. 2024 ) . It leads to a notable performance drop, especially problematic when access to the previous data is unavailable. In this paper, we propose a novel weight regularization method that has advantages of both of those two approaches. Our method, Soft Weight Rescaling (SWR), directly reduces the weight magnitudes close to the initial values by scaling down weights. With a minimal computational overhead, it effectively prevents unbounded weight growth. Unlike previous methods, SWR recovers plasticity without losing information. In addition, our theoretical analysis proves that SWR bounds weight magnitude and balances weight magnitude between layers. We evaluate the effectiveness of SWR on standard image classification benchmarks across various scenarios—including warm-start learning, continual learning, and single-task learning—comparing it with other regularization methods and highlighting its advantages, particularly in the case of VGG-16. The contributions of this work are summarized as follows. First, We introduce a novel method that effectively prevents unbounded weight growth while preserving previously learned information and maintaining network plasticity. Second, we provide a theoretical analysis demonstrating that SWR bounds the magnitude of the weights and balances the weight magnitude across layers without degrading model performance. Finally, we empirically show that SWR improves generalization performance across various learning scenarios. The rest of this paper is organized as follows. Section 2 reviews studies on weight magnitude and regularization methods. In Section 3 , we explain weight rescaling and propose a novel regularization method, Soft Weight Rescaling. Then, in Section 4 , we evaluate the effectiveness of Soft Weight Rescaling by comparing it with other regularization methods across various experimental settings. 2 Related Works Unbounded Weight Growth. There have been studies associated with the weight magnitude. Krogh & Hertz 1991 ; Bartlett 1996 indicated that the magnitude of weights is related to generalization performance. Besides, as the magnitude of the weights increases, the Lipschitz constant also tends to grow ( Couellan 2021 ) . This leads to higher sensitivity of the network, potentially affecting its stability and generalization. Ghiasi et al. 2024 demonstrated that weight decay plays a role in reducing sensitivity for noise. Moreover, Lyle et al. 2024b claimed that unbounded weight growth is one of the factors of plasticity loss in training with non-stationary distribution. These studies indicate that enormous weight magnitudes disturb effective learning. Unfortunately, weight growth is inevitable in deep learning. Neyshabur et al. 2017 showed that when the training error converges to 0, the weight magnitude gets unbounded. Merrill et al. 2020 observed that weight magnitude increases with O(\sqrt{t}) , where t is the update step during transformer training. These explanations highlight the ongoing need for weight regularization in modern deep learning. Weight Regularization. Various methods have been proposed to regularize the weight magnitude. L2 regularization, which is also termed as weight decay, is a method to apply an additional loss term that penalizes the L2 norm of weight. Although it is a method widely used, several studies pointed out its problems ( Ishii & Sato 2018 ; Liu et al. 2021 ) . Yoshida & Miyato 2017 suggested regularizing the spectral norm of the weight matrix and showed improved generalization performance in various experiments. Kumar et al. 2020 regularized the weights to maintain the effective rank of the features. On the other hand, several studies have explored how to utilize the initialized weights.

Read the complete text