Pruning-based Data Selection and Network Fusion for Efficient Deep Learning

Humaira Kousar, Hasnain Irshad Bhatti, Jaekyun Moon · 2025

Read Pruning-based Data Selection and Network Fusion for Efficient Deep Learning by Humaira Kousar, Hasnain Irshad Bhatti, Jaekyun Moon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-01118.txt.

Introduction

Humaira Kousar, Hasnain Bhatti, and Jaekyun Moon pick training data by a cut-down twin of the network itself. Selection methods can cost more than the training they save, and hand labels cost money. Their 2025 method, PruneFuse, prunes the dense network to a small stand-in. The stand-in picks the most informative samples in rounds, at low cost.

Once the samples run rich enough, the gains of the stand-in fuse back into the dense network as a warm start. Tests on a spread of datasets show selection cost down, grades past base methods, and whole training time cut. Small first, then fuse, is the craft they pitch for lean deep learning.

Catalog record

CREATORHumaira Kousar, Hasnain Irshad Bhatti, Jaekyun Moon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-01118

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pruning-based Data Selection and Network Fusion for Efficient Deep Learning Humaira Kousar, Hasnain Irshad Bhatti, Jaekyun Moon Abstract Efficient data selection is essential for improving the training efficiency of deep neural networks and reducing the associated annotation costs. However, traditional methods tend to be computationally expensive, limiting their scalability and real-world applicability. We introduce PruneFuse, a novel method that combines pruning and network fusion to enhance data selection and accelerate network training. In PruneFuse, the original dense network is pruned to generate a smaller surrogate model that efficiently selects the most informative samples from the dataset. Once this iterative data selection selects sufficient samples, the insights learned from the pruned model are seamlessly integrated with the dense model through network fusion, providing an optimized initialization that accelerates training. Extensive experimentation on various datasets demonstrates that PruneFuse significantly reduces computational costs for data selection, achieves better performance than baselines, and accelerates the overall training process. 1 Introduction Deep learning models have achieved remarkable success across various domains, ranging from image recognition to natural language processing [ 1 , 2 , 3 ] . However, the performance of models heavily relies on the access of large amounts of labeled data for training [ 4 ] . In real-world applications, manually annotating massive datasets can be prohibitively expensive and time-consuming. Data selection techniques such as Active Learning (AL) [ 5 ] offer a promising solution to address this challenge by iteratively selecting the most informative samples from the unlabeled dataset for annotation. The goal of AL is to reduce labeling costs while maintaining or improving model performance. However, as data and modal complexity grow, traditional AL techniques that require iterative model training become computationally expensive, limiting scalability in resource-constrained settings. In this paper, we propose PruneFuse, a novel strategy for efficient data selection in active learning setting that overcomes the limitations of traditional approaches. Our approach is based on model pruning, which reduces the complexity of neural networks. By utilizing small pruned networks for data selection, we eliminate the need to train large models during the data selection phase, thus significantly reducing computational demands. Additionally after the data selection phase, we utilize the learning of these pruned networks to train the final model through a fusion process, which harnesses the insights from the trained networks to accelerate convergence and improve the generalization of the final model. Contributions. Our key contribution is to introduce an efficient and rapid data selection technique that leverages pruned networks. By employing pruned networks as data selectors, PruneFuse ensures computationally efficient selection of informative samples leading to overall superior generalization. Furthermore, we propose a novel concept of fusing these pruned networks with the original untrained model, enhancing model initialization and accelerating convergence during training. We demonstrate the broad applicability of PruneFuse across various network architectures, offering a flexible tool for efficient data selection in diverse deep learning settings. Extensive experimentation on CIFAR-10, CIFAR-100, and Tiny-ImageNet-200 shows that PruneFuse achieves superior performance to state-of-the-art active learning methods while significantly reducing computational costs. Figure 1 : Overview of the PruneFuse Method : (1) An untrained neural network is initially pruned to form a structured, pruned network \theta_{p} . (2) This pruned network \theta_{p} queries the dataset to select prime candidates for annotation, similar to active learning techniques. (3) \theta_{p} is then trained on these labeled samples to form the trained pruned network \theta_{p}^{*} . (4) The trained pruned network \theta_{p}^{*} is fused with the base model \theta , resulting in a fused model. (5) The fused model is further trained on a selected subset of the data, incorporating knowledge distillation from \theta_{p}^{*} . 2 Background and Related Works Subset Selection Framework. Active Learning (AL) is widely utilized iterative approach tailored for situations with abundant unlabeled data. Given a classification task with C classes and a large pool of unlabeled samples U , AL revolves around selectively querying the most informative samples from U for labeling. The process commences with an initial set of randomly sampled data s^{0} from U , which is subsequently labeled. In subsequent rounds, AL augments the labeled set L by adding newly identified informative samples. This cycle repeats until a predefined number of labeled samples b are selected. Data Selection. Approaches such as [ 6 , 7 , 5 ] aim to select informative samples using techniques like diversity maximization and Bayesian uncertainty estimation. Parallelly, the domain of active learning has unveiled strategies, such as [ 8 , 9 , 7 , 10 , 11 , 6 ] , which prioritize samples that can maximize information gain, thereby enhancing model performance with minimal labeling effort. While these methods achieve efficient data selection, they still require training large models for the selection process, resulting in significant computational overhead. Other strategies such as [ 12 ] optimize this selection process by matching the gradients of subset with training or validation set based on orthogonal matching algorithm and [ 13 ] performs meta-learning based approach for online data selection. SubSelNet [ 14 ] proposes to approximate a model that can be used to select the subset for various architectures without retraining the target model, hence reducing the overall overhead. However, it involves pre-training routine which is very costly and needed again for any change in data or model distribution. SVP [ 15 ] introduces to use small proxy models for data selection but discards these proxies before training the target model. Additionally, structural discrepancies between the proxy and target models may result in sub-optimal data selections. Our approach also builds on this foundation of using small model (which in our case is a pruned model) but it enables direct integration with the target model through the fusion process. This ensures that the knowledge acquired during data selection is retained and actively contributes to the training of the original model. Also, the architectural coherence between the pruned and the target model provides a more seamless and effective mechanism for data selection, enhancing overall model performance and efficiency. Efficient Deep Learning. Methods such as [ 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] have been proposed to reduce model size and computational requirements. Neural Network pruning has been extensively investigated as a technique to reduce the complexity of deep neural networks [ 18 ] . Pruning strategies can be broadly divided into Unstructured Pruning [ 24 , 25 , 26 , 27 ] and Structured Pruning [ 28 , 29 , 30 , 31 ] based on the granularity and regularity of the pruning scheme. Unstructured pruning often yields a superior accuracy-size trade-off whereas structured pruning offers practical speedup and compression without necessitating specialized hardware [ 32 ] . While pruning literature suggests pruning after training [ 33 ] or during training [ 34 , 35 ] , recent research explore the viability of pruning at initialization [ 36 , 37 , 38 , 37 , 39 ] . In our work, we leverage the benefits of pruning at initialization to create a small representative model for efficient data selection. 3 PruneFuse In this section, we delineate the PruneFuse methodology, illustrated in Fig. 1 (and Algorithm 1 provided in Appendix). The procedure begins with

Read the complete text