Strategic Fusion Optimizes Transformer Compression

Md Shoaibur Rahman · 2025

Read Strategic Fusion Optimizes Transformer Compression by Md Shoaibur Rahman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03273.txt.

Introduction

Md Shoaibur Rahman, a solo author, presents a computing study from 2025 on shrinking large language models. Cutting whole layers from a model saves size and cost, but picking which layers to cut is tricky. Single clues, like layer activity or weight size, each miss part of the picture.

Rahman tested fourteen cutting plans across nine data sets. Twelve plans used one clue each, drawn from signals like attention and gradients. Two fusion plans merged the clues, one by regression and one by random forest. The forest fusion won on seven data sets and nearly won the other two. A teaching step then restored lost accuracy, even beating the base model on six sets. Size-to-accuracy gains averaged near nineteen-fold.

Catalog record

CREATORMd Shoaibur Rahman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-03273

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Strategic Fusion Optimizes Transformer Compression Md Shoaibur Rahman Abstract This study investigates transformer model compression by systematically pruning its layers. We evaluated 14 pruning strategies across nine diverse datasets, including 12 strategies based on different signals obtained from layer activations, mutual information, gradients, weights, and attention. To address the limitations of single-signal strategies, we introduced two fusion strategies, linear regression and random forest, which combine individual strategies (i.e., strategic fusion), for more informed pruning decisions. Additionally, we applied knowledge distillation to mitigate any accuracy loss during layer pruning. Our results reveal that random forest strategic fusion outperforms individual strategies in seven out of nine datasets and achieves near-optimal performance in the other two. The distilled random forest surpasses the original accuracy in six datasets and mitigates accuracy drops in the remaining three. Knowledge distillation also improves the accuracy-to-size ratio by an average factor of 18.84 across all datasets. Supported by mathematical foundations and biological analogies, our findings suggest that strategically combining multiple signals can lead to efficient, high-performing transformer models for resource-constrained applications. [email protected] 1 Introduction Large pre-trained transformer models ( Vaswani et al., 2023 ) have transformed natural language processing by achieving state-of-the-art performance across a wide range of tasks, from sentiment analysis to text classification ( Rahman & Borera, 2024 ) . However, their significant computational and memory requirements present a challenge for deployment in resource-constrained environments, such as edge devices or real-time applications ( Strubell et al., 2019 ) . Model compression has emerged as a crucial area of research to address these limitations, enabling efficient use of these powerful models without compromising their performance. Past studies explored various approaches for compressing large transformer models, including pruning unimportant weights ( Han et al., 2015 ) , quantizing parameters ( Gong et al., 2014 ) , and knowledge distillation ( Hinton et al., 2015 ) . Recent work also extends beyond individual parameters to prune entire layers considered less critical, using strategies such as activation-based ( Ganguli & Chong, 2024 ) , gradient-based ( Molchanov et al., 2017 ; Yang et al., 2023 ) , mutual information-based ( Isik et al., 2022 ) , weight-based ( Frankle & Carbin, 2019 ) , or attention-based ( Michel et al., 2019 ) pruning. Although these techniques effectively reduce model size and computational costs, they often rely on single metrics to determine layer redundancy. However, focusing solely on a single signal, which often requires predefined pruning rules, may not fully capture the nuanced contributions of a layer to downstream tasks ( Hooker et al., 2021 ; Zhang et al., 2022 ) . Therefore, single signal-based pruning strategies often lead to drastic drops in accuracy. Crucially, existing works seldom explore how to combine multiple pruning signals or thoroughly examine how to sequence layer pruning decisions without a predefined rule. This gap underlines the need for a framework that accounts for multiple pruning signals and systematically assesses the importance of each layer to optimize performance trade-offs. In this work, we examined 12 individual pruning strategies using signals from layer activations, mutual information, gradients, weights, and attention, and propose two fusion strategies to integrate these signals. We provided the mathematical and biological intuition behind the choice of each strategy, which demonstrates theoretical and practical perspectives on their selection. We tested our strategies using the BERT model ( Devlin et al., 2019 ) . We evaluated 14 pruning strategies across nine datasets, focusing on text classification and sentiment analysis. Each individual strategy computes a layer-specific metric or signal, identifies layers to prune based on a predefined rule (e.g., based on the min or max value of the metric), and fine-tunes the compressed model. The fusion strategies, based on linear regression and random forest, integrate multiple pruning signals to automatically identify optimal layer pruning schedules without a predefined rule. Finally, we incorporated knowledge distillation-based training, where the compressed model was trained using the original model as a teacher to recover any accuracy lost during pruning. Our experiments reveal that integrating multiple signals through strategic fusion consistently outperforms single-metric approaches in both accuracy improvement and model size reduction. In particular, random forest-based fusion strategy achieves the best performance in seven out of nine datasets, while ranking second and third best for the remaining two datasets. Furthermore, knowledge distillation exceeds the original accuracy for six datasets and mitigates the accuracy drops in three other datasets. The accuracy-to-size ratio after distillation increases by an average factor of 18.84 across all datasets. Our results also highlight that which layers are pruned and in what sequence matters greatly: edge layers often carry critical information, and high-performing strategies automatically learn not to prune them early. Taken together, our findings demonstrate that the fusion of individual strategies into a data-driven framework can lead to an effective and efficient compressed transformer model. 2 Methodology 2.1 Datasets We employed nine diverse text classification datasets of varying domains (e.g., user reviews, scientific abstracts, news articles) and number of labels (2 to 20 classes): newsgroup, dbpedia_14 (dbpedia), arxiv-classification (arxiv), patent-classification (patent), yahoo_answers_topics (yahoo), yelp_review_full (yelp), ag_news (agnews), imdb, and amazon_polarity (amazon). All datasets except newsgroup are available via Hugging Face, whereas newsgroup is accessible through scikit-learn. All input sequences were tokenized using the BERT tokenizer and padded or truncated to a maximum length of 32 tokens for consistent processing across datasets. 2.2 Layer Pruning Strategies 2.2.1 Activation-Based Pruning Activation-based pruning is a natural approach to identify redundant transformer layers due to the fundamental role activations play in neural network operations ( Ganguli & Chong, 2024 ) . Activations, measured as the output of neurons after applying nonlinear transformations, represent the input in a transformed feature space. Layers with specific activation patterns may contribute in various ways to the overall functionality of the network. From a mathematical perspective, activations can be viewed as mappings from the input space to a feature space, where their magnitude and distribution signify the importance of a layer. Layers exhibiting consistently low or sparse activations are hypothesized to contribute minimally to overall feature transformation. Biologically, this aligns with the idea that neurons or brain regions with persistently low firing rates play a negligible role in processing, further motivating the use of activations as a basis for pruning ( Harvey et al., 2013 ) . To quantify activations, we used three strategies to aggregate them into different signals or metrics: inhibition , intensity , and energy . These metrics offer complementary insights into the importance of activations within a layer. Inhibition : Inhibition measures the mean value of activations in a layer: A_{\text{inhibition}}=\frac{1}{n\cdot d}\sum_{i=1}^{n}\sum_{j=1}^{d}A_{i,j} (1) where A\in\mathbb{R}^{n\times d} is the activation matrix for n tokens and d hidden dimensions. Although this metric could also be termed polarity (negative values indicate inhibition, positive values indicate excitation), we found it to be consistently negative across layers and datasets, leading to its designation as inhibition. Although pruning inhibitory layers may occasionally be beneficial, it is often risky. Inhibitory layers may encode critical information, balance representations, or filter noise, ensuring efficient processing. Biologically, inhibitory neurons regulate excitatory activity, maintaining stability ( Znamenskiy et al., 2024 ) . As a result, inhibition may not always be a reliable metric for pruning. Intensity : Intensity measures the mean of absolute

Read the complete text