SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models

Xun Liang Hanyu Wang Huayi Lai Simin Niu Shichao Song, Jiawei Yang Jihao Zhao Feiyu Xiong Bo Tang Zhiyu Li · 2025

Read SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models by Xun Liang Hanyu Wang Huayi Lai Simin Niu Shichao Song, Jiawei Yang Jihao Zhao Feiyu Xiong Bo Tang Zhiyu Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07605.txt.

Introduction

Shrinking a large language model for one job, without retraining it, is the goal of a method called SEAP in this 2025 paper by Xun Liang and co-authors. Big models spend most of their compute on parts a given task never uses. SEAP spots the activation patterns a task relies on, treats those parts as its experts, and prunes the rest.

At half the model pruned away, SEAP beat two rival methods by over twenty points on the study's measures. At a gentler twenty percent pruning, it gave up only 2.2 percent of the full model's performance. No training runs were needed at all. The authors present the method as a cheap way to slim models for real deployment.

Catalog record

CREATORXun Liang Hanyu Wang Huayi Lai Simin Niu Shichao Song, Jiawei Yang Jihao Zhao Feiyu Xiong Bo Tang Zhiyu Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-07605

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models Xun Liang Hanyu Wang Huayi Lai Simin Niu Shichao Song, Jiawei Yang Jihao Zhao Feiyu Xiong Bo Tang Zhiyu Li Abstract Large Language Models have achieved remarkable success across various natural language processing tasks, yet their high computational cost during inference remains a major bottleneck. This paper introduces Sparse Expert Activation Pruning (SEAP) 1 1 1 Our code is available at https://github.com/IAAR-Shanghai/SEAP , a training-free pruning method that selectively retains task-relevant parameters to reduce inference overhead. Inspired by the clustering patterns of hidden states and activations in LLMs, SEAP identifies task-specific expert activation patterns and prunes the model while preserving task performance and enhancing computational efficiency. Experimental results demonstrate that SEAP significantly reduces computational overhead while maintaining competitive accuracy. Notably, at 50% pruning, SEAP surpasses both WandA and FLAP by over 20%, and at 20% pruning, it incurs only a 2.2% performance drop compared to the dense model. These findings highlight SEAP’s scalability and effectiveness, making it a promising approach for optimizing large-scale LLMs. 1 Introduction Large Language Models (LLMs) have achieved remarkable success across a wide spectrum of natural language processing (NLP) tasks Zhao et al. 2024 ; Zheng et al. 2025 , demonstrating their versatility and adaptability in diverse applications. However, their deployment in real-world scenarios remains a significant challenge due to the substantial computational demands during inference. The inference process of LLMs is constrained by memory bandwidth and hardware limitations Chavan et al. 2024 , making efficient deployment particularly difficult, especially in resource-constrained environments such as real-time systems and edge computing. As LLMs continue to scale, these challenges become even more pronounced, necessitating novel approaches to optimize computational efficiency while preserving model performance. To mitigate the computational overhead of LLMs, several techniques have been explored. Quantization methods Bai et al. 2021 ; Frantar et al. 2023 reduce weight precision, while Mixture of Experts (MoE) architectures Shazeer et al. 2017 ; Lepikhin et al. 2020 ; Fedus et al. 2022 dynamically activate only subsets of the network to improve efficiency. Another widely adopted approach is pruning Frantar and Alistarh 2023 ; Ma et al. 2023 ; Liu et al. 2024 , which removes redundant parameters, neurons, or connections to reduce inference costs and storage requirements. Despite the effectiveness of pruning in reducing model complexity, most existing methods are static, relying on activation distributions collected from general datasets such as WikiText-2 Merity et al. 2016 and C4 Raffel et al. 2020a . These methods apply a uniform pruning strategy across all tasks, which may lead to suboptimal efficiency and fail to fully leverage task-specific knowledge requirements. Inspired by cognitive neuroscience, where different brain regions are selectively activated based on task demands, we hypothesize that a similar mechanism exists in LLMs—where different tasks rely on distinct sets of neurons working collaboratively. This suggests that pruning strategies should be adaptive rather than static, dynamically selecting the most relevant parameters for each task. By leveraging task-specific activation patterns, we can develop a more effective sparsification technique that maintains task performance while significantly enhancing computational efficiency. Figure 1: Visualization of hidden states h(P) from different tasks. Each point represents the activation of a hidden state in the model for a specific task. The clustering patterns illustrate how tasks with similar requirements tend to activate similar regions in the model. Motivation Discovery In cognitive neuroscience, the brain parcellation theory posits that different regions of the brain are selectively activated based on specific task demands, thereby optimizing cognitive efficiency Mesulam 2000 ; Li et al. 2022 . Inspired by this principle, we investigate whether a similar mechanism exists in LLMs — where distinct tasks may activate different sets of neurons, forming task-specific computational pathways. This perspective challenges conventional pruning approaches, which typically apply a uniform sparsity pattern across all tasks, potentially overlooking task-dependent knowledge representations. We hypothesize that the knowledge requirements and activation patterns of different tasks are closely linked. If so, pruning should not be a one-size-fits-all process but rather a dynamic, task-aware strategy. By leveraging this relationship, pruning can adaptively retain the most relevant parameters based on each task’s specific characteristics, thereby enhancing computational efficiency while preserving task performance. To validate this hypothesis, we design a multi-task experiment to analyze whether different tasks induce partitioned representations in LLM hidden states. We select seven task categories, covering a broad spectrum of linguistic and reasoning challenges, including common sense reasoning, mathematical problem-solving, and scientific question answering. We construct task-specific knowledge corpora consisting of question-answer pairs and feed them into an LLM to extract hidden states across multiple layers. The details of the corpus construction process are provided in Section A . To visualize the structure of these hidden states, we project their high-dimensional representations onto a two-dimensional plane. As shown in Figure 1 , embeddings are initially intermingled. However, as forward propagation progresses, the model refines the semantic features of the input, leading to increasingly distinct task-specific clusters. For instance, in the final layer, GSM8K Cobbe et al. 2021 , a challenging mathematical reasoning task, exhibits a clear separation from common-sense reasoning tasks. Similarly, OBQA Mihaylov et al. 2018 and ARC Clark et al. 2018 , both of which rely heavily on external scientific knowledge, form a closely related distribution. On the other hand, PIQA Bisk et al. 2020 and HellaSwag Zellers et al. 2019 , though both categorized as common-sense reasoning tasks, emphasize everyday knowledge, positioning them below OBQA and ARC in the visualization. Interestingly, Winogrande Sakaguchi et al. 2019 , a pronoun resolution task, clusters with certain HellaSwag prompts, likely due to their shared reliance on pronoun-based reasoning. Lastly, BoolQ Clark et al. 2019 , a contextual reasoning task, forms a distinct grouping, indicating its unique reliance on contextual comprehension. Figure 2: Framework of the SEAP approach. The left side shows the Motivation Discovery phase, where task-specific activation patterns are identified by analyzing hidden states and neuron activations extracted from the task corpus. The right side illustrates the Training-free Sparse Expert Activation Pruning process, consisting of five main steps described in Section 2.1 . These findings suggest that each task occupies a distinct region within the hidden state space, with certain dimensions of activation corresponding to task-specific information. This observation draws an intriguing parallel to the functional specialization of the human brain, where different cognitive processes activate distinct neural circuits. Building on this insight, we propose our central hypothesis: During inference, leveraging task-specific activation patterns and dynamically selecting the most relevant parameters can significantly reduce computational overhead while maintaining task performance. This task-adaptive pruning paradigm stands in contrast to traditional static pruning approaches, offering a promising path toward more efficient and specialized LLM deployment. Contributions Our key contributions are: • We analyze task-specific activation patterns in LLMs, revealing their correlation with hidden state distributions and providing new insights for adaptive sparsification. • We propose SEAP, a training-free, task-adaptive pruning method that dynamically adjusts sparsity based on task type, improving efficiency while preserving performance. • We demonstrate that SEAP outperforms existing baselines in task accuracy, storage efficiency, and inference speed, confirming its effectiveness for efficient LLM deployment. 2 Method 2.1 Overview of SEAP Building on the insights from Section 1 ,

Read the complete text