Less Is More? Examining Fairness in Pruned Large Language Models for Summarising Opinions

Nannan Huang, Haytham Fayek, Xiuzhen Zhang · 2025

Read Less Is More? Examining Fairness in Pruned Large Language Models for Summarising Opinions by Nannan Huang, Haytham Fayek, Xiuzhen Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17610.txt.

Introduction

Nannan Huang, Haytham Fayek, and Xiuzhen Zhang ask whether shrinking language models also skews their fairness in a 2025 paper. Pruning trims a model's size cheaply, and opinion summaries shape public views, so hidden bias shifts matter. No one had mapped how pruning choices move fairness.

Across three pruning methods, several calibration sets, and three open models, the pruning method itself moved fairness scores far more than the calibration data did. Guided by that insight, the team designed a pruning rule that cuts weights with strong output influence but weak input use. Their method held or improved fairness where rivals degraded it, across four fairness metrics. Human judges agreed the summaries were fairer. Code is public.

Catalog record

CREATORNannan Huang, Haytham Fayek, Xiuzhen Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17610

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Less Is More? Examining Fairness in Pruned Large Language Models for Summarising Opinions Nannan Huang, Haytham Fayek, Xiuzhen Zhang Abstract Model compression through post-training pruning offers a way to reduce model size and computational requirements without significantly impacting model performance. However, the effect of pruning on the fairness of LLM-generated summaries remains unexplored, particularly for opinion summarisation where biased outputs could influence public views. In this paper, we present a comprehensive empirical analysis of opinion summarisation, examining three state-of-the-art pruning methods and various calibration sets across three open-source LLMs using four fairness metrics. Our systematic analysis reveals that pruning methods have a greater impact on fairness than calibration sets. Building on these insights, we propose High Gradient Low Activation (HGLA) pruning, which identifies and removes parameters that are redundant for input processing but influential in output generation. Our experiments demonstrate that HGLA can better maintain or even improve fairness compared to existing methods, showing promise across models and tasks where traditional methods have limitations. Our human evaluation shows HGLA-generated outputs are fairer than existing state-of-the-art pruning methods. Code is available at: https://github.com/amberhuang01/HGLA . 1 Introduction Scaling language models has led to emergent capabilities Brown 2020 ; Radford et al. 2019 ; Chowdhery et al. 2023 ; Touvron et al. 2023 ; Le Scao et al. 2023 , however, it has also led to increased computational resources Wu et al. 2022 ; Zhu et al. 2023 ; Menghani 2023 . While reducing the size of LLMs through post-training pruning has become popular and attracted substantial research interest Frantar and Alistarh 2023 ; Sun et al. 2023 ; Das et al. 2023 , their impact beyond performance metrics remains largely understudied, with the work by Chrysostomou et al. 2024 on model hallucination being one of the only studies that investigates pruning from a non-performance angle. Another critical but unexplored aspect of pruning is its impact on model fairness. The use of AI language systems with inherent biases could shape the way audiences interpret and process information Jakesch et al. 2023 ; Durmus et al. 2023 ; Epstein et al. 2023 , especially in tasks such as opinion summarisation where biased outputs could significantly influence public opinion. It is well-acknowledged that LLMs were exposed to uncurated data that may contain societal bias, which inevitably perpetuates social stereotypes in models Vig et al. 2020 ; Sheng et al. 2019 ; Liang et al. 2021 ; Gallegos et al. 2024 ; Li et al. 2023 and propagates to downstream tasks Feng et al. 2023 . Post-training pruning typically uses a small calibration set of samples to identify parameters for removal. Given the fairness concerns, it is crucial to examine all aspects of post-training pruning procedures that could impact model fairness, including both pruning methods and the calibration set. While recent work has shown that the choice of calibration set significantly impacts model performance Williams and Aletras 2024 , its effect on model fairness remains unexplored. This gap is particularly concerning given the established relationship between training data and model bias. To the best of our knowledge, our work presents the first systematic investigation of how pruning affects fairness in LLM-generated summaries. Through a comprehensive analysis using three state-of-the-art pruning methods, various calibration sets across three large language models, and four fairness metrics, our results demonstrate that pruning methods have a greater influence on model fairness than calibration sets. We find that pruning more model parameters can lead to decreased fairness while maintaining performance, suggesting that performance-focused pruning methods may inadvertently amplify biases. This complex relationship varies across tasks, pruning methods, and calibration sets, highlighting the critical importance of carefully selecting pruning methods and continuously monitoring fairness metrics during the pruning process. In response to these limitations, we introduce High Gradient Low Activation (HGLA) pruning using calibration sets—a novel procedure that identifies and removes parameters that are redundant in input processing (i.e., low activation) but influential in output generation (i.e., high gradient), offering a promising method to maintain or even improve model fairness during the post-training pruning process. Our human evaluation study demonstrates that this pruning procedure produces fairer summaries compared to other state-of-the-art post-pruning methods. 2 Related Work 2.1 Post-training Pruning for LLMs Model pruning aims to reduce model size while maintaining performance. Magnitude pruning examines the absolute value of model weights as a simple baseline. The activation magnitudes indicate how parameters contribute to processing inputs. By analysing activation strengths using calibration data, we can identify less crucial parameters, such as those showing consistently weak activations that may be unnecessary and redundant in processing input. Recent post-training pruning methods such as SparseGPT Frantar and Alistarh 2023 and Wanda Sun et al. 2023 are mainly guided using activation information to prune such redundant parameters. A model’s gradients provide information on how sensitive its output is with respect to its parameters. Parameters with larger gradients are more sensitive on the model’s output, as small changes to these parameters would significantly change the output. GBLM-Pruner Das et al. 2023 combines the information in both activation and gradient with a scaling factor to decide how much contribution is made by the gradient. Parameters that produce the lowest gradient and activation should be removed since removing these parameters has the minimum impact on both the output of the model and also how the model processes input information. These methods require calibration data, typically sampled from C4 Raffel et al. 2020 . While research shows calibration set selection significantly impacts model performance Williams and Aletras 2024 , its impact on model fairness has not been studied. While LLMs can maintain performance with up to 50% pruning Jaiswal et al. 2023 , pruning’s effect on fairness remains unexplored. Prior work has shown pruning can reduce hallucination by increasing source document reliance Chrysostomou et al. 2024 but the impact on model fairness has not been studied. The key contributions of this work are: (1) evaluating fairness across multiple pruning methods, (2) examining effects of different calibration sets, and (3) based on the limitations identified in existing methods, we propose a novel pruning procedure that addresses these challenges. 2.2 Fairness in Summarising Opinions Prior work has explored social bias in language models across gender, race, and other attributes Vig et al. 2020 ; Sheng et al. 2019 ; Liang et al. 2021 ; Ladhak et al. 2023 ; Santurkar et al. 2023 , showing how these biases propagate from training data to downstream tasks Feng et al. 2023 . In opinion summarisation, fair models should represent diverse opinions proportionally to their source documents Shandilya et al. 2018 , and biased outputs risk misrepresenting input distributions. Prior research has examined fairness across demographic attributes such as gender, race, political orientation Dash et al. 2019 , dialect Blodgett et al. 2016 , and opinion diversity Huang et al. 2023 . Recent studies have investigated fairness in fine-tuning Huang et al. 2024 and prompt-based models Zhang et al. 2023 . However, the impact of post-training pruning on fairness in abstractive summarisation remains unexplored. 3 High Gradient Low Activation Pruning (HGLA) Formally, pruning methods assign a significance score \mathbf{S}_{i,j} to each element of a layer’s weight matrix \mathbf{W}_{i,j} . State-of-the-art post-training pruning methods can also factor in additional

Read the complete text