Adaptive Pruning of Pretrained Transformer via Differential Inclusions
Yizhuo Ding, Ke Fan, Yikai Wang, Xinwei Sun, Yanwei FuSchool of Data Science, Fudan University {yzding22,kfan21}@m.fudan.edu.cn ; [email protected] ; {sunxinwei,yanweifu}@fudan.edu.cn · 2025
Read Adaptive Pruning of Pretrained Transformer via Differential Inclusions by Yizhuo Ding, Ke Fan, Yikai Wang, Xinwei Sun, Yanwei FuSchool of Data Science, Fudan University {yzding22,kfan21}@m.fudan.edu.cn ; [email protected] ; {sunxinwei,yanweifu}@fudan.edu.cn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03289.txt.
Introduction
Slimming a giant transformer usually means repeating the whole pruning job for every target size, which burns computing time. Yizhuo Ding and co-authors prune once for all sizes in this 2025 paper. Their method follows a solution path: as a mask dial turns, weights drop out in a principled order. The surviving pattern at any dial setting gives a ready model at that sparsity.
Transformers are split into paired modules to make the path well-behaved. One pruning stage then yields a whole family of models, from lightly trimmed to heavily cut. Users can pick their size after the fact, not before. Cheaper inference for large models is the prize. Flexibility also suits deployment across devices of very different power.
Catalog record
| CREATOR | Yizhuo Ding, Ke Fan, Yikai Wang, Xinwei Sun, Yanwei FuSchool of Data Science, Fudan University {yzding22,kfan21}@m.fudan.edu.cn ; [email protected] ; {sunxinwei,yanweifu}@fudan.edu.cn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03289
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adaptive Pruning of Pretrained Transformer via Differential Inclusions Yizhuo Ding, Ke Fan, Yikai Wang, Xinwei Sun, Yanwei FuSchool of Data Science, Fudan University {yzding22,kfan21}@m.fudan.edu.cn ; [email protected] ; {sunxinwei,yanweifu}@fudan.edu.cn Abstract Large transformers have demonstrated remarkable success, making it necessary to compress these models to reduce inference costs while preserving their performance. Current compression algorithms prune transformers at fixed compression ratios, requiring a unique pruning process for each ratio, which results in high computational costs. In contrast, we propose pruning of pretrained transformers at any desired ratio within a single pruning stage, based on a differential inclusion for a mask parameter. This dynamic can generate the whole regularization solution path of the mask parameter, whose support set identifies the network structure. Therefore, the solution path identifies a Transformer weight family with various sparsity levels, offering greater flexibility and customization. In this paper, we introduce such an effective pruning method, termed SPP (Solution Path Pruning). To achieve effective pruning, we segment the transformers into paired modules, including query-key pairs, value-projection pairs, and sequential linear layers, and apply low-rank compression to these pairs, maintaining the output structure while enabling structural compression within the inner states. Extensive experiments conducted on various well-known transformer backbones have demonstrated the efficacy of SPP. Our code is available at https://github.com/yizhuoDi/Solution-Path-Pruning. † † footnotetext: Corresponding authors. 1 Introduction Transformers have succeeded in various tasks due to their scalability, parallel processing abilities, and capacity to learn complex data patterns. Pretrained transformers, in particular, perform well on downstream tasks by leveraging large datasets during training. These models are highly effective for transfer learning, allowing fine-tuning for different tasks and delivering strong performance in many natural language processing applications. However, their large size, often with billions of parameters, makes them difficult to deploy on low-cost hardware. Despite their widespread use and impressive performance ( Radford et al. 2021 ; Touvron et al. 2021a ) , running transformers on lightweight devices like phones and laptops remains challenging due to computational constraints. Therefore, compressing transformers to run efficiently on affordable hardware is crucial. Various techniques have been developed to compress transformers while preserving their performance. These include weight sharing ( Lan et al. 2019 ) , low-rank factorization ( Yu et al. 2017 ) , quantization ( Gong et al. 2014 ; Polino et al. 2018 ; Tao et al. 2022 ) , knowledge distillation ( Hinton et al. 2015 ; Yuan et al. 2019 ; Touvron et al. 2020 ; Liu et al. 2020 ) , and pruning ( Yu & Xiang 2023 ; Yang et al. 2023 ; Shi et al. 2023 ; Yin et al. 2023 ) . Pruning methods typically involve structured pruning, wherein entire neurons or attention heads are removed based on importance, followed by fine-tuning to regain accuracy. Despite the array of pruning methods developed for transformers, current algorithms face a fundamental challenge: they are tailored to achieve a predefined pruning ratio by the end of the pruning phase. Figure 1: Comparison of SPP and lasso method. (a) SPP can obtain sparse models of all sparsity after search stage , which includes update stage and prune stage. (b) Lasso method can only obtain one sparse model in a single search stage. Typically, when targeting a new degree of sparsity, the entire pruning process is restarted to meet the new target. Restarting such an entire pruning process for each level of sparsity introduces high costs for model compression. For over-parameterized models, the training cost can be substantial. To address this challenge, we propose an adaptive compression strategy through a differential inclusion for mask-based pruning of pretrained transformer, named SPP (Solution Path Pruning). The dynamics efficiently generates a whole regularization solution path of masks,where each mask’s support set captures the important weights. As is shown in Figure 1 , along this solution trajectory, the sparsity of the projected target model incrementally increases, with important weights learned earlier in the dynamics until convergent to the fully dense network. Besides, this dynamics has a simple iterative form to implement. Therefore, after running a whole iteration, we can obtain a transformer weight family with different compression ratios. Note that Fu et al. 2020 used a similar pruning method for structured sparsity in neural networks trained from scratch. They also used differential inclusions with inverse scale spaces to train the network. Our focus, however, is on pre-trained transformers, which are much harder to prune while maintaining performance. Unlike common pruning methods, SPP does not require restarting the search stage, rendering it a cost-effective pruning technique requiring just one search stage to derive a transformer weight family with diverse sparsity levels from the uncompressed model. Our exploration of transformer structures adopts a novel and efficient approach, automatically learning compression structures throughout training, thus enjoying greater flexibility and customization. Currently, no other algorithm can theoretically guarantee optimization convergence while producing sparse models with different levels of sparsity during training. For example, the Upop ( Shi et al. 2023 ) algorithm only makes the model structure sparse at the final step, with the mask becoming zero only at that point. The OFA ( Cai et al. 2019 ) method lacks a convergence guarantee and simply fixes the mask updates during training to achieve a sparse network. SPP addresses this by using the solution path to obtain sparse structures without affecting convergence, providing a stronger theoretical foundation. Notably, our method can be extended to prune large language models (LLMs) with one-shot post training. In this paper, we applied SPP to the classification task dataset ImageNet ( Deng et al. 2009 ) using the DeiT ( Touvron et al. 2021a ) backbone. Furthermore, we also extended the method to image and text retrieval datasets, using CLIP models ( Radford et al. 2021 ) . Our contributions are summarized as follows: • We develop a differential inclusion-based method for adaptive compression of pretrained transformer, enabling the acquisition of sparse models with different sparsity within a single search stage, significantly reducing the cost of model pruning. • We introduce the novel concept of the Transformer Weight Family, obtained through a simple iterative algorithm following the discretization of the differential inclusion. • We also prove the global convergence of the method in such a non-convex optimization based on the Kurdyka-Łojasiewicz framework, demonstrating that the entire iterative sequence converges to a critical point of the empirical loss function from arbitrary initializations. • We demonstrate the effectiveness of our framework across various backbones and datasets. Results show that we can significantly reduce the computational costs while preserving the prediction accuracy. 2 Related work Transformer pruning. In the expanding field of Transformer-based model research, the concept of pruning Transformer has garnered considerable interest for its potential to streamline model architectures. Over recent years, various structured pruning methods have been developed. ViT-Slim ( Chavan et al. 2022 ) employs a single-shot training approach to search for optimal sub-structures in vision transformers. SAViT ( Chen et al. 2021 ) collaboratively prunes all components of Vision Transformers, integrating structure-aware interactions and using a Taylor-based optimization function for joint importance. UP-ViTs ( Yu & Wu 2023 ) introduces a unified pruning framework for vision transformers, focusing on comprehensive pruning of all components of ViT models and their variants. The process