Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning
Richa Upadhyay [email protected], Ronald Phlypo [email protected], Rajkumar Saini [email protected], Marcus Liwicki [email protected] · 2025
Read Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning by Richa Upadhyay [email protected], Ronald Phlypo [email protected], Rajkumar Saini [email protected], Marcus Liwicki [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12115.txt.
Introduction
Richa Upadhyay and co-authors teach multi-task networks to find their own sparse shapes in a paper from two thousand twenty-five. Sparse networks drop unneeded weights, but the right sparsity level is usually set by hand for each task. Their framework learns that level during training.
The method, Meta-Sparsity, borrows from meta-learning, where a model learns how to adapt fast. It applies channel-wise sparsity penalties during meta-training so shared structure across tasks stays lean. Tests ran on two data sets spanning pixel-level and image-level tasks. The approach held up well across many tasks, the authors report, and adapted to unseen ones. They pitch it as a step toward leaner, more flexible networks.
Catalog record
| CREATOR | Richa Upadhyay [email protected], Ronald Phlypo [email protected], Rajkumar Saini [email protected], Marcus Liwicki [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 126 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12115
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning Richa Upadhyay [email protected], Ronald Phlypo [email protected], Rajkumar Saini [email protected], Marcus Liwicki [email protected] Abstract This paper presents meta-sparsity, a framework for learning model sparsity, basically learning the parameter that controls the degree of sparsity, that allows deep neural networks (DNNs) to inherently generate optimal sparse shared structures in multi-task learning (MTL) setting. This proposed approach enables the dynamic learning of sparsity patterns across a variety of tasks, unlike traditional sparsity methods that rely heavily on manual hyperparameter tuning. Inspired by Model Agnostic Meta-Learning (MAML), the emphasis is on learning shared and optimally sparse parameters in multi-task scenarios by implementing a penalty-based, channel-wise structured sparsity during the meta-training phase. This method improves the model’s efficacy by removing unnecessary parameters and enhances its ability to handle both seen and previously unseen tasks. The effectiveness of meta-sparsity is rigorously evaluated by extensive experiments on two datasets, NYU-v2 and CelebAMask-HQ, covering a broad spectrum of tasks ranging from pixel-level to image-level predictions. The results show that the proposed approach performs well across many tasks, indicating its potential as a versatile tool for creating efficient and adaptable sparse neural networks. This work, therefore, presents an approach towards learning sparsity, contributing to the efforts in the field of sparse neural networks and suggesting new directions for research towards parsimonious models. 1 Introduction Model compression in DL ( DL ) is a process that aims at reducing the size and complexity of neural network models while maintaining their performance. In the current technological trend, model compression has become essential in DL due to its role in enabling complex neural networks to operate efficiently on devices with constrained resources by substantially improving memory usage and computation requirements. It facilitates the practical application of DL models in real-world scenarios, fostering sustainability and enhancing accessibility on a broader scale ( Deng et al. (2020) ). It is possible to reduce the size and complexity of a model by leveraging techniques such as Neural Architecture Search (NAS)( Ren et al. (2021) ), model value quantization, model distillation ( Hinton et al. (2015) ), low-rank factorization ( Sainath et al. (2013) ), parameter sharing ( Desai & Shrivastava (2023) ), sparsification ( Hoefler et al. (2021) ), and many more. This work focuses on two of these techniques, i.e., parameter sharing in the form of hard parameter sharing in multi-task learning and sparsification. However, our emphasis extends beyond simply compressing models. We combine the concept of model sparsification with MTL ( MTL ), viewing sparsification as a technique for selecting optimally shared features in a multi-task setting. This methodology enables the strategic distribution of these features across diverse tasks during MTL , enhancing the joint learning process. Sparsification 1 1 1 Note: In the context of this work, sparsification should not be confused with pruning , which is also a widely used term in model compression. This is because both are different concepts; pruning is a technique that can help to achieve sparsification by systematically removing certain elements; sparsification is a much broader concept that can be reached through various methods, including but not limited to pruning. So the words sparsification and pruning are not used interchangeably. in DL pertains to transforming a densely connected neural network or a dense model to a sparse one, where a considerable proportion of the parameters (typically model weights) are set to zero. This reduction of active weights leads to less computational load and memory usage during both the training and inference phases. Additionally, sparsification not only highlights the importance of key features (sparsity-driven feature selection) but also significantly enhances the model’s ability to generalize to new data ( Hoefler et al. (2021) ). When implementing sparsity in DNN ( DNN ), three key aspects must be considered: (i) identifying what elements should be made sparse, (ii) determining when to induce sparsity, and (iii) deciding on the method i.e., how to achieve sparsity. The article by Hoefler et al. (2021) offers a comprehensive survey that thoroughly explores these factors related to sparsity, among various other elements. Several elements can be sparsified in a DNN , like the parameters (or weights), neurons, filters in convolution layers, and heads in attention layers. Sparsification can be scheduled post-training, during training, or as sparse training, as illustrated in Figure 1 . Further details on these methods are provided in the subsequent text. Various techniques can be employed to attain sparsity, including methods like magnitude pruning based on thresholds, sparsification based on input-output sensitivity, penalty-driven approaches, variational methods, and many more. In this work, we emphasize sparsifying the model parameters during the training by applying penalty-based (regularization-based) sparsification methods. Figure 1 : This figure illustrates (a-d) a few common approaches to achieving sparse models and (e) the proposed approach of meta-sparsification. Various sparsity-inducing strategies have been developed to facilitate the transformation of a dense model into a sparse model ( Hoefler et al. (2021) ). However, a vital question that arises before implementing these strategies is determining when and how to apply sparsification. Figure 1 broadly illustrates some of these sparsity scheduling techniques as discussed by Hubens (2020) . The one-shot sparsification (Figure 1 (a)) is a fundamental technique that sparsifies an already trained dense model until a desired sparsity is achieved and then fine-tunes the sparse model. Fine-tuning allows the model to relearn. It helps to re-distribute the weights across the remaining parts of the network, ensuring that performance does not suffer significantly. Figure 1 (b) shows iterative sparsification where sparsity is induced gradually over many steps ( Han et al. (2015) ; Liu et al. (2017) ). Only a handful of weights are eliminated at each iteration, and the network is fine-tuned to restore performance. This process is repeated multiple times. Zhu & Gupta (2017) presented the concept of automated progressive (or gradual) sparsification (Figure 1 (c)) in their work, which is very similar to iterative sparsification. While both methods incrementally introduce sparsity into a dense network, automated gradual sparsification is different because it sparsifies the weights during the training of the model, allegedly claiming that it eliminates the necessity for subsequent fine-tuning. Figure 1 (d) illustrates sparse training, where a sparse model is trained, and while training, it tries to retain the sparsity. This is often referred to as sparse-to-sparse training ( Dettmers & Zettlemoyer (2019) ; Evci et al. (2020) ), while the others Figure 1 (a-c) are referred to as dense to sparse training. In general, any sparsification involves the use of a few hyper-parameters that control the degree of sparsity, like thresholds in magnitude pruning, sparsification budget (or target), the regularization parameter in penalty-based sparsity approach, the dropout rate (probability), learning rate in gradual pruning, and several others. Often, approaches like grid search, random search, heuristic methods, etc, are employed to determine the optimal values of these crucial hyper-parameters. We introduce a method to learn these hyper-parameters that induce sparsity through meta-learning. This work, therefore, proposes meta-sparsification, broadly depicted in Figure 1 (e). In this work, we refer to it as learning to learn sparsity , or learned sparsity , or meta-sparsity . To be more precise, it is learning to learn or meta-learning ( Finn