Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution
Qiao Xiao, Alan Ansell, Boqian Wu, Lu Yin, Mykola Pechenizkiy, Shiwei Liu, Decebal Constantin Mocanu · 2025
Read Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution by Qiao Xiao, Alan Ansell, Boqian Wu, Lu Yin, Mykola Pechenizkiy, Shiwei Liu, Decebal Constantin Mocanu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24037.txt.
Introduction
Qiao Xiao and co-authors published a computing paper in 2025 on fine-tuning large language models that have been pruned to a sparse form. Sparse models deploy cheaply, but adapting them to new tasks is hard. Older fine-tuning either adds dense parts back or freezes the sparse layout. Their method lets the sparse structure itself shift during training.
The method, called SEFT, moves sparse updates to where they help and revives pruned weights when useful. Importance guides how the layout adapts, keeping the efficiency gains. Tests on three model families across benchmarks beat rival methods on score, memory, and time. The code is public.
Catalog record
| CREATOR | Qiao Xiao, Alan Ansell, Boqian Wu, Lu Yin, Mykola Pechenizkiy, Shiwei Liu, Decebal Constantin Mocanu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24037
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution Qiao Xiao, Alan Ansell, Boqian Wu, Lu Yin, Mykola Pechenizkiy, Shiwei Liu, Decebal Constantin Mocanu Abstract Sparse large language models (LLMs) offer an attractive direction toward efficient deployment, but adapting them to downstream tasks remains challenging. The central difficulty is to enable effective task adaptation without sacrificing the efficiency advantages of sparsity. Existing fine-tuning methods are not well-suited to this setting, as they either introduce additional dense parameters or assume a fixed sparse topology, limiting their compatibility with sparse LLMs. In this paper, we propose Sparsity Evolution Fine-Tuning (SEFT) , a fine-tuning framework designed specifically for sparse LLMs. SEFT allows sparse structure to evolve during fine-tuning by periodically reallocating sparse task-specific updates and reactivating previously pruned weights when beneficial. At the same time, SEFT preserves the efficiency advantages of sparsity through topology adaptation based on parameter importance. Experiments on LLaMA, DeepSeek, and Mistral models across multiple benchmarks show that SEFT delivers stronger performance while offering superior memory and time efficiency compared to existing baselines. Our code is publicly available at: https://github.com/QiaoXiao7282/SEFT . 1 Introduction Large language models (LLMs) have achieved remarkable success across a wide range of tasks, including language understanding ( Zhu et al., 2023 ) , reasoning ( Wei et al., 2022 ) , and code generation ( Vaithilingam et al., 2022 ) . However, their massive parameter counts create substantial challenges for real-world deployment, especially in memory-constrained and latency-sensitive settings. As a result, improving the efficiency of LLMs has become a central problem in both research and practice. Among existing efficiency techniques, post-training pruning offers an attractive path for compressing LLMs without the prohibitive cost of retraining from scratch ( Ma et al., 2023 ; Zhu et al., 2024 ) . Recent methods such as SparseGPT ( Frantar and Alistarh, 2023 ) and Wanda ( Sun et al., 2023 ) can produce highly sparse models while preserving much of the original performance. Yet pruning alone is not sufficient: as sparsity increases, performance degradation becomes increasingly pronounced, especially in high-sparsity regimes (e.g., \geq 60\% ( Lu et al., 2024a ) ). More importantly, post-training pruning typically produces a task-agnostic sparse structure, determined before downstream adaptation. A sparse topology that appears effective under calibration data may still remove connections that are important for a specific downstream task ( Bandari et al., 2024 ; Williams et al., 2026 ) . This exposes a fundamental limitation in current adaptation methods on sparse LLMs: the challenge is not only to preserve the efficiency benefits of sparsity, but also to adapt sparse models to downstream tasks when the pruned topology itself may be misaligned with task-specific needs. However, existing methods do not fully address this challenge. General parameter-efficient fine-tuning (PEFT) methods, such as LoRA and its variants ( Hu et al., 2021 ; Xu et al., 2023 ; Dettmers et al., 2024 ) , are effective for adapting dense LLMs, but are not well suited to sparse deployment. As they either introduce additional storage overhead for adapters or produce dense models after merging, thereby undermining the efficiency benefits of sparsity, as illustrated in Figure 1 (a). More recent sparsity-preserved fine-tuning methods ( Lu et al., 2024b ; Munoz et al., 2024 ; Hu et al., 2025 ) maintain sparsity throughout adaptation, but typically update only a predefined set of sparse parameters. This restriction can lead to suboptimal performance when the resulting sparse topology is not well aligned with the downstream task. As a result, current methods either fail to preserve sparse efficiency or do not support the sparse topology itself to adapt. This raises a key question: can we efficiently fine-tune sparse LLMs while allowing their sparse updates to evolve for better adaptation to downstream tasks? Figure 1: Comparison of LoRA and SEFT fine-tuning for sparse LLMs. While LoRA-based methods require low-rank matrices, SEFT utilizes indices (arrows) and corresponding deltas (green squares) to update LLM parameters. To answer this question, we propose Sparsity Evolution Fine-Tuning (SEFT) , a sparse fine-tuning framework for adapting post-pruning sparse LLMs. The core idea of SEFT is to allow sparse task-specific updates evolve with downstream tasks in both their locations and mask constraints. First, the locations of sparse task-specific updates are periodically refreshed during fine-tuning. Second, these update locations are not restricted to the initial pruning mask, allowing previously pruned weights to be explored and reactivated when beneficial. Together, these properties enable SEFT to better align sparse LLMs with downstream tasks, as illustrated in Figure 1 (b). To realize this efficiently, SEFT consists of two coupled components. A lightweight Delta Support Update governs the evolution of sparse task-specific updates by refreshing where they are allocated, while Sparse Topology Adaptation preserves the efficiency advantages of sparsity by enforcing the target sparsity level on the model. Unlike LoRA-based approaches that introduce auxiliary adapters, SEFT directly updates the model while keeping the base model sparse. We evaluate SEFT on a diverse set of sparse LLMs from the LLaMA ( Touvron et al., 2023a ; Touvron et al., 2023b ; Dubey et al., 2024 ) , DeepSeek ( Bi et al., 2024 ) , and Mistral ( Jiang et al., 2023 ) families. Across model scales from 7B to 13B and a range of downstream benchmarks, including commonsense reasoning, MMLU, and GSM8K, SEFT consistently delivers better performance while also offering favorable memory usage and training efficiency. Our contributions in this work can be summarized as follows: • We propose SEFT, an efficient fine-tuning approach tailored to sparse LLMs, which dynamically evolves the sparse topology to enable task-specific adaptation. • We introduce a sparsity adaptation mechanism based on a sensitivity-guided pruning criterion, which preserves the target sparsity level without sacrificing performance. • We demonstrate that SEFT offers favorable memory usage and training efficiency compared with prior sparsity-preserving fine-tuning methods. • We conduct comprehensive experiments across multiple sparse LLM architectures (LLaMA, DeepSeek, and Mistral) and tasks (commonsense reasoning, MMLU, and GSM8K), validating SEFT’s generality and effectiveness in both general-purpose and task-specific settings. 2 Preliminaries 2.1 Parameter-Efficient Fine-Tuning (PEFT) Parameter-efficient fine-tuning (PEFT) has emerged as a popular approach to adapt LLMs to downstream tasks while significantly reducing the computational and memory overhead associated with traditional dense full fine-tuning ( Pfeiffer et al., 2023 ; Ding et al., 2023 ; Han et al., 2024 ) . Instead of updating all parameters of a pre-trained model, PEFT methods introduce additional trainable parameters, which typically are a small fraction of the original model, while keeping the base model weights frozen. This significantly reduces computational overhead during fine-tuning. One widely used PEFT method is Low-Rank Adaptation (LoRA) ( Hu et al., 2021 ) . In LoRA, low-rank matrices are introduced into specific layers (e.g., attention or feedforward layers) to capture task-specific adaptations. Specifically, let \mathbf{W}\in\mathbb{R}^{d\times d} represent a frozen weight matrix in a pre-trained model. LoRA introduces a low-rank decomposition: \mathbf{W}^{\prime}=\mathbf{W}+\Delta\mathbf{W},\quad\Delta\mathbf{W}=\mathbf{AB} (1) where \mathbf{A}\in\mathbb{R}^{d\times r} and \mathbf{B}\in\mathbb{R}^{r\times d} are low-rank matrices (r\ll d) , and only \mathbf{A} and \mathbf{B} are trainable. While PEFT methods like LoRA have proven effective for dense models, they are inherently not well-suited for sparse LLMs, as they either require dense updates or fail to preserve sparsity after fine-tuning. 2.2 Sparse