Restoring Neural Network Plasticity for Faster Transfer Learning

Xander Coetzer, Arné Schreuder, Anna Sergeevna Bosman · 2026

Read Restoring Neural Network Plasticity for Faster Transfer Learning by Xander Coetzer, Arné Schreuder, Anna Sergeevna Bosman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20860.txt.

Introduction

Coetzer and co-authors study a simple fix for lost plasticity in transfer learning. Models pretrained on ImageNet can have saturated weights that barely change during fine tuning. This can block them from fitting a new task well. Their fix resets chosen weights before fine tuning starts. They test the idea on convolutional networks and vision transformers. Both model types gain accuracy and converge faster on several image tasks. The method adds almost no compute cost. It also fits standard transfer learning pipelines.

Catalog record

CREATORXander Coetzer, Arné Schreuder, Anna Sergeevna Bosman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37209
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20860

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Restoring Neural Network Plasticity for Faster Transfer Learning Xander Coetzer, Arné Schreuder, Anna Sergeevna Bosman Abstract Transfer learning with models pretrained on ImageNet has become a standard practice in computer vision. Transfer learning refers to fine-tuning pretrained weights of a neural network on a downstream task, typically unrelated to ImageNet. However, pretrained weights can become saturated and may yield insignificant gradients, failing to adapt to the downstream task. This hinders the ability of the model to train effectively, and is commonly referred to as loss of neural plasticity. Loss of plasticity may prevent the model from fully adapting to the target domain, especially when the downstream dataset is atypical in nature. While this issue has been widely explored in continual learning, it remains relatively understudied in the context of transfer learning. In this work, we propose the use of a targeted weight re-initialization strategy to restore neural plasticity prior to fine-tuning. Our experiments show that both convolutional neural networks (CNNs) and vision transformers (ViTs) benefit from this approach, yielding higher test accuracy with faster convergence on several image classification benchmarks. Our method introduces negligible computational overhead and is compatible with common transfer learning pipelines. Keywords Transfer Learning \cdot Computer Vision \cdot Plasticity 1 Introduction Transfer learning has become a cornerstone of modern computer vision workflows, largely due to the widespread availability of powerful deep learning models pretrained on large-scale datasets such as ImageNet ( 8 ) . In typical scenarios, these pretrained models are fine-tuned on a downstream dataset that often differs substantially in size, structure, or domain compared to the dataset used for pretraining. While the transfer learning approach offers significant gains in both performance and data efficiency, it also introduces critical challenges that remain underexplored. One such challenge is the loss of neural plasticity, i.e., the reduced ability of a neural network to adapt to new data distributions during fine-tuning. This issue arises when certain pretrained weights, particularly those with near-zero magnitudes, result in negligible gradients during backpropagation ( 13 ) . Such weights become effectively inactive, hindering learning and preventing the model from fully adapting to the target domain. Although loss of neural plasticity has been extensively studied in the context of continual learning ( 6 ; 14 ) , it has received limited attention in transfer learning, where similar dynamics can emerge due to the abrupt domain shift between pretraining and fine-tuning datasets ( 15 ) . Recent research in continual learning proposed a selective weight reinitialization strategy to restore plasticity by resetting weights deemed unimportant or inactive ( 12 ) . Inspired by this line of work, our research investigates the application of weight reinitialization techniques within the transfer learning context. We selectively reinitialize a subset of low utility pretrained weights prior to fine-tuning to restore their capacity for learning. We evaluate the effectiveness of this approach across multiple architectures, including convolutional neural networks (CNNs) and vision transformers (ViTs), on a range of datasets of varying size and complexity. Novel contributions of this work are summarized as follows: • We establish the relationship between neural plasticity and transfer learning. • Neural plasticity is restored prior to fine-tuning through a targeted weight reinitialization strategy, which identifies and reinitializes inactive weights. • The proposed approach improves downstream classification accuracy, especially in tasks involving substantial domain shifts. • Further, our method enhances training efficiency by enabling faster convergence, without increasing model size or requiring architectural modifications. By addressing a critical but underexplored aspect of transfer learning, our work provides a principled way to improve model adaptation and performance on diverse downstream tasks. This method is lightweight, easily integrated into existing pipelines, and generalizes well across model types and datasets. The rest of the paper is structured as follows: Section 2 discusses the essential background topics and related work, Section 3 details the proposed methodology and experimental setup. Section 4 presents the experimental results, and Section 5 critically analyses the findings. Finally, Section 6 concludes the paper and offers some directions for future research. 2 Background and Related Work To contextualize the proposed approach, the following related topics are discussed in this section: transfer learning (Section 2.1 ), continual learning and neural plasticity (Section 2.2 ), weight reinitialization (Section 2.3 ), and parameter efficient fine-tuning (Section 2.4 ). 2.1 Transfer Learning Transfer learning is a technique that aims to transfer knowledge from one model to another ( 21 ) . Practically, this involves using the weights that have been learned by the former model as a starting point for training the new model ( 8 ) . It is especially useful in areas where the amount of available, labeled training data is limited. The performance of the new model scales with the size of the source dataset as well as the similarity between the source and target datasets ( 20 ) . The transition from the source distribution to the target distribution can cause loss of neural plasticity ( 15 ) , similar to what is observed in continual learning. 2.2 Continual Learning and Neural Plasticity Continual learning refers to the scenario in which a model is trained on a dynamic data distribution, i.e., the data distribution changes as training progresses. Continual learning in neural networks is enabled via neural plasticity and memory stability. The former refers to the ability of the model to adapt well to new data, while the latter refers to the ability of the model to remain performant on old data ( 25 ) . Transfer learning has no need for memory stability, but is similar to continual learning as far as neural plasticity is concerned. Plasticity is a term adopted from neuroscience, and refers to the ability of a neural network model to adapt its weights when trained on new data which the model has not encountered before ( 14 ) . Loss of plasticity occurs when a model being trained on new data cannot “overwrite” previously learned patterns to properly fit the new distribution ( 13 ) . Dohare et al. ( 6 ) introduce three main causes of loss of plasticity in continual learning. The first is an increase in constant or dead units, which occurs when gradients become insignificant and weights are no longer updated to effectively fit the downstream task. The second is a steady growth of the average weight magnitude, as larger weights often lead to slower training. Lastly, loss of plasticity can be caused by a drop in the effective rank of the representation. Such drop occurs when the output is determined by a subset of the weights in the network, rather than each weight having a more or less equal influence. This paper will focus on addressing the first of these causes, i.e., non-contributing weights. Since small weights are less likely to significantly impact the output of the network ( 12 ) , we focus our study on pretrained weights close to zero. 2.3 Reinitialization Algorithms Reinitialization algorithms have been effectively applied to increase generalization performance in neural networks and maintain plasticity in the context of continual learning ( 6 ; 19 ; 22 ) . Most notably, Hernandez-Garcia et al. ( 12 ) proposed a selective weight reinitialization approach, focusing on modifying specific weights within a

Read the complete text