EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning

Songlin Zhao, Michael Pitts, Zhuwei Qin · 2025

Read EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning by Songlin Zhao, Michael Pitts, Zhuwei Qin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19935.txt.

Introduction

Songlin Zhao, Michael Pitts, and Zhuwei Qin present EfficientXpert, a way to specialize large language models cheaply. Low-rank adaptation cuts training cost but leaves a dense model, while pruning done apart can remove weights that adaptation later needs. The aim is a sparse expert that still runs on limited hardware.

EfficientXpert adapts the low-rank updates and the sparse weight set together. Its ForeSight Mask scores weights through a reconstruction stand-in that uses the changing adapted weights, and a Partial Brain Surgeon step realigns the adapter in closed form. At high sparsity, the best variant keeps nearly all dense-adaptation performance on the tested models, with modest extra training time and memory. The study also guides how much adapter recovery capacity to choose.

Catalog record

CREATORSonglin Zhao, Michael Pitts, Zhuwei Qin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19935

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning Songlin Zhao, Michael Pitts, Zhuwei Qin Abstract The rapid advancement of large language models (LLMs) has created an increasing demand for domain-specialized variants in areas such as law, healthcare, and finance. However, their large size remains a barrier to deployment in resource-constrained environments, and existing compression methods either fail to generalize across domains or introduce high overhead. In this work, we propose EfficientXpert , a lightweight domain pruning framework that combines a propagation-aware pruning criterion (Foresight Mask) with an efficient adapter update algorithm (Partial Brain Surgeon). Embedded within the LoRA fine-tuning process, EfficientXpert enables a one-step transformation of general pretrained models into sparse, domain-adapted experts. It achieves up to 98% of original dense model performance at 40% sparsity on tasks in the health and legal domains, outperforming state-of-the-art approaches. Further analysis reveals significant domain-dependent structural shifts that undermine general pruning masks, highlighting the necessity of adaptive, domain-aware pruning strategies tailored to each domain’s unique model changes. Introduction Domain-specific LLMs have advanced rapidly in recent years, driven by the growing demand for high accuracy and deep contextual understanding in specialized domains such as legal, healthcare, and finance applications ( 21 ; 32 ; 13 ) . While general-purpose LLMs demonstrate broad capabilities, they often underperform on tasks requiring domain-specific knowledge and reasoning ( 20 ) . To address this gap and efficiently adapt models to domain-specific vocabulary, structure, and inference tasks, Parameter-Efficient Fine-Tuning (PEFT) is frequently employed, achieving performance comparable to full-parameter fine-tuning ( 21 ; 15 ) . Among various PEFT methods, Low-Rank Adaptation (LoRA) has become particularly popular due to its ability to achieve strong adaptation by modifying only a small subset of parameters through low-rank updates ( 11 ) . Despite reducing training costs, PEFT-adapted domain-specific LLMs still inherit the large size and inference overhead of their base models, posing significant challenges for deployment in resource-constrained environments. Consequently, there is a critical need to compress domain-specialized LLMs, retaining domain performance while minimizing computational and memory demands ( 8 ; 10 ) . One promising approach to reducing the size of LLMs is model pruning, which eliminates redundant weights to lower memory and computation costs ( 10 ) . Recent methods such as SparseGPT ( 8 ) and Wanda ( 23 ) demonstrate that a two-step pipeline, pruning a general model and subsequently applying PEFT, can preserve performance on general-purpose benchmarks, even at high sparsity levels. However, such general-purpose pruning strategies often fail in domain-specific settings due to two key limitations. First, they rely on static pruning masks that fail to reflect the shifts in weight importance and distribution that arise during domain adaptation ( 31 ; 4 ) . Second, PEFT methods like LoRA do not preserve sparsity: reapplying the pruning mask afterward often leads to large performance drops due to misalignment ( 31 ; 11 ) . To address these challenges, recent methods such as ATP ( 16 ) and D-Pruner ( 31 ) incorporate domain-aware pruning and fine-tuning strategies to bridge this gap. ATP employs a trainable pruning-decision generator, which adaptively refines model structure throughout the LoRA fine-tuning process ( 16 ) . D-Pruner, in contrast, uses a dual-pruning strategy that jointly considers both general and domain-specific weight importance using gradient-based approximations and regularization techniques ( 31 ) . While effective, their reliance on gradient-based methods with retraining and auxiliary neural networks incurs substantial computational overhead, adding further cost to an already expensive fine-tuning pipeline ( 31 ; 16 ) . Moreover, these prior methods evaluate only a limited set of tasks, whereas our broader evaluation demonstrates that pruning sensitivity depends more critically on domain than task, highlighting a key oversight in earlier approaches. In this work, we introduce EfficientXpert, a lightweight and efficient framework that unifies pruning and LoRA fine-tuning to obtain domain-specific LLMs. By explicitly accounting for domain-specific pruning sensitivity, EfficientXpert provides a scalable, low-cost solution ideal for real-world use. It introduces two key innovations: • Foresight Mask: A domain-aware pruning method that incorporates forward error propagation into its scoring mechanism by estimating the impact of weights on downstream representations. It enables dynamic, gradient-free mask updates during fine-tuning by leveraging both frozen base weights and evolving adapters. • Partial Brain Surgeon: An efficient adapter realignment step that solves a ridge regression to suppress pruned coordinates, ensuring that low-rank updates stay aligned with the evolving sparse structure. Together, EfficientXpert integrates both components into the LoRA fine-tuning process, enabling a one-step transformation of a general pretrained model into a sparse domain expert with minimal overhead. Across a comprehensive set of tasks that span the health and legal domains, EfficientXpert achieves up to 99.8% and 98.43% of the dense model’s performance at 40% sparsity on LLaMA-7B. Beyond empirical results, our analysis demonstrates that pruning sensitivity is fundamentally shaped by the domain of adaptation. Some domains exhibit substantial shifts in weight importance and outlier structure, causing strong misalignment with general-purpose pruning masks. Additionally, we find that domain shifts lead to greater divergence in adapter subspaces than task differences. These findings confirm that domain, more than task, drives pruning behavior and highlight the need for domain-adaptive pruning in efficient model specialization. Related Work Parameter-Efficient Fine-Tuning As LLMs grow in size and capability, Parameter-Efficient Fine-Tuning (PEFT) has emerged as a vital strategy for adapting pretrained models to domains and downstream tasks ( 15 ) . Among these, Low-Rank Adaptation (LoRA) has emerged as one of the most widely adopted approaches for adapting LLMs to new domains and tasks. LoRA injects trainable low-rank matrices into the attention and feedforward layers of transformer models, enabling efficient adaptation while leaving the original weights unchanged ( 11 ) . LoRA’s simplicity and strong empirical performance have driven its rapid adoption in domains such as biomedicine, law, and finance, where recent models like FinGPT ( 28 ) , Med-PaLM ( 2 ) , and LawGPT ( 33 ) successfully adapt general-purpose LLMs to specialized tasks with limited supervision and compute. Despite its success, LoRA does not reduce the size or latency of the final model, since the frozen backbone remains fully intact. Recent works have begun to explore combinations of LoRA with pruning on general tasks, seeking to preserve LoRA’s training efficiency while improving deployment efficiency ( 30 ; 29 ) . However, the mechanisms by which low-rank updates capture domain- and task-specific information remain largely unexplored. Figure 1: (a) Comparison of EfficientXpert with existing domain pruning methods. (b) Overview of EfficientXpert framework, including ForeSight Mask and Partial Brain Surgeon. EfficientXpert iteratively updates adapters, smooths importance scores, applies corrections to surviving weights, and merges the mask into dense weights to create a sparse, domain-specialized expert. Neural Network Pruning Pruning has long been a fundamental approach for compressing neural networks by eliminating weights with minimal impact on task performance. While iterative magnitude-based pruning proved effective in early CNNs ( 10 ; 19 ; 18 ) , it becomes impractical for LLMs due to high retraining costs and activation outliers ( 22 ; 1 ; 14 ; 5 ) . To address this, general pruning methods like SparseGPT ( 8 ) and Wanda ( 23 ) compress pretrained LLMs by identifying and removing weights that contribute

Read the complete text