Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning
Xin Ning, Qiankun Li, Xiaolong Huang, Qiupu Chen, Feng He, Weijun Li, Prayag Tiwari, Xinwang Liu · 2026
Read Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning by Xin Ning, Qiankun Li, Xiaolong Huang, Qiupu Chen, Feng He, Weijun Li, Prayag Tiwari, Xinwang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22838.txt.
Introduction
Ning and seven colleagues propose DualOpt, an optimizer that treats training from scratch and fine-tuning as different jobs instead of using one rule for both. From scratch, it applies layer-wise weight decay in real time to aid convergence and generalization. For fine-tuning, it adds weight rollback so upstream and downstream weight patterns stay consistent, limiting forgotten knowledge. Across image classification, detection, and segmentation tasks, DualOpt reaches state-of-the-art results.
Catalog record
| CREATOR | Xin Ning, Qiankun Li, Xiaolong Huang, Qiupu Chen, Feng He, Weijun Li, Prayag Tiwari, Xinwang Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67344 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22838
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning Xin Ning, Qiankun Li, Xiaolong Huang, Qiupu Chen, Feng He, Weijun Li, Prayag Tiwari, Xinwang Liu Abstract With the accumulation of resources in the era of big data and the rise of pre-trained models in deep learning, optimizing neural networks for various tasks often involves different strategies for fine-tuning pre-trained models versus training from scratch. However, existing optimizers primarily focus on reducing the loss function by updating model parameters, without fully addressing the unique demands of these two major paradigms. In this paper, we propose DualOpt, a novel approach that decouples optimization techniques specifically tailored for these distinct training scenarios. For training from scratch, we introduce real-time layer-wise weight decay, designed to enhance both convergence and generalization by aligning with the characteristics of weight updates and network architecture. For more importantly fine-tuning, we integrate weight rollback with the optimizer, incorporating a rollback term into each weight update step. This ensures consistency in the weight distribution between upstream and downstream models, effectively mitigating knowledge forgetting and improving fine-tuning performance. Additionally, we extend the layer-wise weight decay to dynamically adjust the rollback levels across layers, adapting to the varying demands of different downstream tasks. Extensive experiments across diverse tasks, including image classification, object detection, semantic segmentation, and instance segmentation, demonstrate the broad applicability and state-of-the-art performance of DualOpt. Code is available at https://github.com/qklee-lz/OLOR-AAAI-2024. I Introduction Early deep learning models were typically trained from scratch, and model parameters were initialized randomly [ 35 , 22 ] . However, with the rapid growth of data and advances in deep learning [ 59 , 58 , 60 ] , a wealth of publicly available pre-trained models has emerged [ 51 , 17 , 2 ] . These pre-trained models have significantly enhanced performance in various tasks through transfer learning and fine-tuning techniques [ 46 , 69 ] . Consequently, fine-tuning has become the mainstream approach in many deep learning applications [ 36 , 34 , 77 ] , though training from scratch is still essential for certain tasks and specific scenarios [ 37 , 1 ] . Traditional neural network optimizers focus primarily on minimizing the loss function by updating model parameters [ 70 , 27 ] . In addition, weight decay is often combined with optimization algorithms to constrain the model weights, helping to promote better generalization [ 16 , 14 ] . While these methods are effective, they often overlook the distinct needs of training paradigms. Training from scratch requires careful balancing to ensure efficient convergence and robust generalization, whereas fine-tuning faces the challenge of knowledge forgetting, where the model may lose valuable information learned during pre-training [ 6 ] . Fig. 1: Overview of the proposed DualOpt. The optimization is both fine-tuning with pre-trained weights and training from scratch with random initialization. The DualOpt integrates a weight rollback mechanism for fine-tuning and layer-wise weight decay for training from scratch, providing flexibility and efficiency for various tasks. In the deep learning domain, optimizers like Stochastic Gradient Descent (SGD) [ 24 ] and Adaptive Moment Estimation (Adam) [ 25 ] are widely used. Researchers have continually proposed improvements to enhance optimization efficiency [ 13 , 62 ] , focusing on gradient propagation [ 57 ] , parameter update mechanisms [ 12 ] , and regularization techniques [ 30 ] . Momentum was introduced to accelerate convergence by incorporating past gradient information [ 57 ] . Adaptive Gradient assigned individual learning rates to parameters [ 12 ] , a concept further refined in the Adam optimizer [ 25 ] . Weight decay was introduced to prevent overfitting and improve generalization [ 30 ] , prompting a series of widely used regularization techniques [ 5 , 44 , 15 ] . However, these methods often overlook the structural characteristics of neural networks. When training from scratch, shallow layers capture low-level features (e.g., color, edges, texture), with less risk of overfitting, while deeper layers focus on high-level features (e.g., objects, semantics), which are more prone to overfitting. On the other hand, fine-tuning requires more preserving the general shallow layer information acquired during the pre-training phase. Therefore, the same decay rate across the entire network results in a coarse decay effect. In addition, existing fine-tuning methods have not fully considered the design of optimizers. Replay-based methods [ 54 , 56 , 42 , 47 ] focus on incorporating pre-trained data, but this approach is often inefficient. EWC [ 26 ] and L2-SP [ 71 ] use regularization to constrain weight parameters, but they are even incompatible with adaptive optimizers [ 25 , 44 , 15 ] . Parameter isolation methods [ 23 , 61 ] , which tailor new branches or modules for downstream tasks and networks, introduce additional training parameters and generally lack versatility. In this paper, we propose DualOpt, a novel optimizer that reimagines neural network optimization by decoupling the strategies for fine-tuning and training from scratch. The overview is shown in Fig. 1 . For training from scratch, DualOpt introduces a real-time layer-wise weight decay mechanism that dynamically adjusts the decay rate across different layers of the network. This method enhances convergence speed and generalization by tailoring the optimization process to the structural and functional characteristics of the network. For fine-tuning, DualOpt introduces a weight rollback term to the weight update term, allowing the model to gradually approach the pre-trained weights while adapting to downstream tasks. This mitigates knowledge forgetting and improves fine-tuning performance. In addition, we extend this approach by employing a layer-wise penalty mechanism with penalty decay and a diversified decay rate. Penalty decay integrates feature pyramids and transfer learning, applying more significant rollback to shallow layers that capture features like color and texture, and less rollback to deeper layers that focus on semantic features. The diversified decay rate adjusts rollback levels based on the similarity between the pre-training and downstream task domains, enabling more adaptive and task-specific fine-tuning. We validate the effectiveness of our proposed DualOpt through extensive experiments on tasks including image classification [ 29 , 48 , 67 , 28 , 74 , 49 , 66 , 33 ] , object detection [ 40 ] , semantic segmentation [ 75 ] , and instance segmentation [ 40 ] on 10 popular datasets. The results demonstrate that DualOpt achieves state-of-the-art performance and offers a generalizable solution for a wide range of training scenarios. The contributions of this paper are summarized as follows: • We propose DualOpt, an optimizer that decouples strategies for training from scratch and fine-tuning to address distinct needs of these two major training paradigms. • A real-time layer-wise weight decay mechanism is introduced to enhance convergence and generalization during training from scratch. • For fine-tuning, we propose a weight rollback mechanism combined with layer-wise penalty decay, mitigating knowledge forgetting and improving task adaptation. • The proposed DualOpt achieves state-of-the-art performance on ten popular datasets across various tasks. This work is an extension of our earlier conference version presented at AAAI 2024 [ 22 ] . Compared to the conference version, the journal version includes the following key enhancements: • The previous fine-tuning optimizer is significantly improved with detailed theoretical proofs, efficiency analyses, validation experiments, and comprehensive ablation studies.