One-cycle Structured Pruning with Stability Driven Structure Search
Deepak Ghimire, Dayoung Kil, Seonghwan Jeong, Jaesik Park, Seong-heum Kim · 2025
Read One-cycle Structured Pruning with Stability Driven Structure Search by Deepak Ghimire, Dayoung Kil, Seonghwan Jeong, Jaesik Park, Seong-heum Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13439.txt.
Introduction
Deepak Ghimire and four co-authors prune neural networks in a single training cycle guided by stability, in a paper from 2025. Standard pruning needs several costly stages, while pruning before training often hurts accuracy. Their goal is one pass that keeps top accuracy at low cost.
The method finds a good subnetwork early, using group importance scores and sparsity pressure. A novel signal watches when the chosen subnetwork stops changing across epochs, marking a stable moment to prune. Group sparsity also speeds convergence. Across image benchmarks and three network families, the method matches the best accuracy while ranking among the cheapest to train.
Catalog record
| CREATOR | Deepak Ghimire, Dayoung Kil, Seonghwan Jeong, Jaesik Park, Seong-heum Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-13439
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One-cycle Structured Pruning with Stability Driven Structure Search Deepak Ghimire, Dayoung Kil, Seonghwan Jeong, Jaesik Park, Seong-heum Kim Abstract Existing structured pruning typically involves multi-stage training procedures that often demand heavy computation. Pruning at initialization, which aims to address this limitation, reduces training costs but struggles with performance. To address these challenges, we propose an efficient framework for one-cycle structured pruning without compromising model performance. In this approach, we integrate pre-training, pruning, and fine-tuning into a single training cycle, referred to as the ‘one cycle approach’. The core idea is to search for the optimal sub-network during the early stages of network training, guided by norm-based group saliency criteria and structured sparsity regularization. We introduce a novel pruning indicator that determines the stable pruning epoch by assessing the similarity between evolving pruning sub-networks across consecutive training epochs. Also, group sparsity regularization helps to accelerate the pruning process and results in speeding up the entire process. Extensive experiments on datasets, including CIFAR-10/100, and ImageNet, using VGGNet, ResNet, MobileNet, and ViT architectures, demonstrate that our method achieves state-of-the-art accuracy while being one of the most efficient pruning frameworks in terms of training time. The source code will be made publicly available. 1 Introduction Over the past decade, deep neural networks (DNNs) have demonstrated superior performance at the expense of substantial memory usage, computing power, and energy consumption He et al. 2016 ; Deng et al. 2020 ; Zhai et al. 2022 ; Ghimire et al. 2022 . For compressing and deploying these complex models on low-capability devices, pruning has emerged as a particularly promising approach for many embedded and general-purpose computing platforms. Structured pruning Li et al. 2017 ; Wang et al. 2021 ; Chen et al. 2021 ; Shen et al. 2022 ; Fang et al. 2023 ; Chen et al. 2023 and unstructured pruning Han et al. 2015 ; Frankle and Carbin 2019 ; Ding et al. 2019b ; Sanh et al. 2020 represent the two predominant methods in pruning. Also, there has been increasing interest in middle-level pruning granularity Lin et al. 2022b ; Zhang et al. 2022b ; Park et al. 2023 , which provides the capability to achieve fine-grained structured sparsity combining the advantages of both unstructured fine-grained sparsity and structured coarse-grained sparsity. Figure 1 : Pruning ResNet50 on the ImageNet dataset. Note that the size of the marker represents the training speedup relative to the baseline training cost. Figure 2 : Overview of the OCSPruner algorithm. The process begins with training the baseline model from scratch. As training progresses, from t_{\text{sl-start}} epoch, the pruning sub-structure gradually stabilizes as indicated by the sub-network stability score computed over consecutive training epochs. Final pruning occurs at the stable epoch t^{*} , followed by the remaining training epochs to converge the pruned structure. With the emergence of large-scale DNN models Brown et al. 2020 ; OpenAI 2023 ; Zhai et al. 2022 , there is a growing need to reassess the conventional pruning approach of pre-training, pruning, and fine-tuning, which demands excessive resources. Conventional pruning often requires multi-stage pipelines with repeated fine-tuning procedures or offline re-training. Without efficient learning mechanisms, it is not only a waste of computational power and time but also less applicable to real-time scenarios. One-cycle training and pruning, also known as one-cycle pruning, presents itself as a viable alternative solution to this problem. Obviously, it is important to explore whether a one-cycle pruning framework can rapidly learn new tasks or adapt to new environments in various real-world applications, such as autonomous vehicles, drones, robots, and so on Ghafourian et al. 2024 ; Chen et al. 2021 . To address the limitations of conventional pruning, we introduce O ne- C ycle S tructured Pruner ( OCSPruner ), which eliminates the requirement for pre-trained base network for pruning. Considering scenarios where networks need to learn from scratch within limited resource budgets, the efficiency gained from avoiding pre-training the base model becomes more compelling. Hence, we focus on efficient and stable structured pruning, ensuring model performance in a more sustainable way. Basically, structured pruning offers the advantage of creating slimmer yet structured networks that do not rely on specific hardware or software; however, it can occasionally lead to performance drops, particularly when dealing with high pruning ratios. Considering the insights from the Lottery Ticket Hypothesis Frankle and Carbin 2019 , which suggests that sparse, trainable sub-networks can be identified within larger networks, the timing and method of pruning become critical. Our focus on one-cycle structured pruning involves pruning the network at initialization Frankle and Carbin 2019 ; Lee et al. 2019 ; de Jorge et al. 2021 . However, this type of method may suffer from potential sub-optimal network structures that result in performance degradation. Thus, rather than pruning at initialization, our proposed strategy involves early-phase pruning during training while still maintaining a one-cycle approach. Despite related methods like pruning-aware training (PaT) Shen et al. 2022 and loss-aware automatic selection of pruning criteria (LAASP) Ghimire et al. 2023 , they overlook structural parameter grouping during saliency estimation, leading to reduced performance Fang et al. 2023 . Specifically, PaT Shen et al. 2022 focuses solely on total channel numbers, not considering individual channel identities during stable sub-network selection, while LAASP Ghimire et al. 2023 lacks stability estimation criteria and involves complex iterative pruning. This paper addresses these limitations and introduces additional algorithmic enhancements for early-phase structured pruning during training from scratch, resulting in state-of-the-art performance. Fig. 2 illustrates the overall flow of the proposed pruning algorithm. Our algorithm begins with network training from scratch over a few epochs. Following this, we applied structured sparsity regularization targeting pruning groups of parameters alongside network training. For each training epoch, we proposed a novel method to measure the stability of network pruning, leveraging the similarity among temporarily pruned sub-structures across consecutive training epochs. The sub-structures are determined by globally partitioning the structurally grouped parameters into pruning and non-pruning groups utilizing norm-based group saliency scores. As training progresses, the penalty factor for regularization progressively increases, thereby promoting the partitioned group of pruning parameters to approach zero gradually. Finally, the proposed algorithm automatically determines the epoch of stable pruning, during which final pruning is executed. This will occur when the partitioning of parameters into pruning and non-pruning groups achieves stability across successive training epochs. Following this, standard training will persist with the pruned sub-structure for the remaining epochs. This approach leads to an efficient one-cycle training framework, achieving a fully trained pruned network starting from the randomly initialized base network. The main contributions of this work are summarized as follows: • We proposed an efficient structured pruning algorithm that combines traditional training, pruning, and fine-tuning steps into a single training cycle through the pruning while training approach. • A robust pruning paradigm is suggested, which continuously monitors the evolving pruning sub-structure guided by a pruning stability indicator to achieve an optimal pruning architecture during the early stages of network training. • The effectiveness of the proposed algorithm is validated in various datasets utilizing single-branch VGGNet, multi-branch ResNet, and compact MobileNet architectures. Notably, it achieves 75.49% top-1 and 92.63% top-5 accuracy for the ResNet50 model on the ImageNet dataset, demonstrating a 1.38\times training speedup compared to