A Systematic Study of Compression Ordering for Large Language Models Journal: Engineering Applications of Artificial Intelligence
Shivansh Chhawri, Rahul Mahadik, Suparna Rooj · 2025
Read A Systematic Study of Compression Ordering for Large Language Models Journal: Engineering Applications of Artificial Intelligence by Shivansh Chhawri, Rahul Mahadik, Suparna Rooj (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19495.txt.
Introduction
Shivansh Chhawri, Rahul Mahadik, and Suparna Rooj test whether the order of compression steps changes how well a large language model survives shrinking. Distillation, structured pruning, and low-bit quantization each squeeze a model, but their combined effects were unclear. The test bed is the Qwen2.5 model at three billion parameters.
Pipelines are scored on perplexity, judged quality, clarity, prompt fit, and compression ratio. Quantization alone compresses most, while pruning costs moderate quality. Order matters a great deal: pruning, then distillation, then quantization gives the best balance, shrinking the model by more than three and a half times while keeping instruction following strong.
Catalog record
| CREATOR | Shivansh Chhawri, Rahul Mahadik, Suparna Rooj |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-19495
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Systematic Study of Compression Ordering for Large Language Models Journal: Engineering Applications of Artificial Intelligence Shivansh Chhawri, Rahul Mahadik, Suparna Rooj Abstract Large Language Models (LLMs) require substantial computational resources, making model compression essential for efficient deployment in constrained environments. Among the dominant compression techniques—knowledge distillation, structured pruning, and low-bit quantization—their individual effects are well studied, but their interactions and optimal sequencing remain unclear. This work systematically examines how these techniques perform both independently and in combination when applied to the Qwen2.5-3B model. We evaluate multiple compression pipelines, including single, and proposed three-technique sequences, using perplexity, G-Eval, clarity, prompt alignment, and compression ratio as metrics. Our experiments show that quantization provides the greatest standalone compression, while pruning introduces moderate quality degradation. Critically, the ordering of techniques significantly affects the final model quality: the sequence Pruning → Knowledge Distillation → Quantization (P-KD-Q) yields the best balance, achieving a 3.68 × 3.68\times compression ratio while preserving strong instruction-following and language understanding capabilities. Conversely, pipelines applying quantization early suffer severe performance degradation due to irreversible information loss that impairs subsequent training. Overall, this study offers practical insight into designing effective, ordering-aware compression pipelines for deploying LLMs in resource-limited settings. 1 Introduction Over the past decade, artificial intelligence has experienced an extraordinary surge in capability and adoption, evolving from traditional machine learning methods to powerful deep learning systems and, more recently, to advanced generative models. This rapid growth has been fueled by breakthroughs in neural network architectures, large-scale computing, and the availability of massive datasets. Today, AI is profoundly transforming diverse domains—including healthcare [ 40 ] , finance [ 5 ] , manufacturing [ 34 ] , defense, etc. Despite remarkable progress in AI, machines do not inherently understand or generate human language without sophisticated computational models. Language modeling [ 7 ] , a core task in natural language processing, focuses on predicting the next token—word or character—within a sequence, enabling systems to capture linguistic structure and contextual meaning which before the emergence of AI was a critical bottleneck [ 23 ] . Large language models (LLMs) have rapidly grown in popularity due to their ability to perform a wide range of tasks such as text generation, reasoning, summarization after the publication of the attention based transformer model [ 44 ] research paper, which is the base of all the LLMs. The emergence of LLMs, image generation systems, and diffusion-based techniques highlights the accelerating pace of innovation, marking a significant shift in the way intelligent systems are developed and deployed. Research on large language models has advanced rapidly since the introduction of transformer-based architectures, beginning with OpenAI’s Generative Pre-trained Transformer (GPT) series, where GPT [ 35 ] , GPT-2 [ 36 ] and GPT-3 [ 4 ] demonstrated that scaling the size of the model and training data significantly improve language understanding and generation. This paradigm was further expanded by Google’s BERT [ 11 ] and T5 [ 37 ] , which highlighted the effectiveness of bidirectional and sequence-to-sequence pretraining strategies. Meta’s LLaMA models [ 43 ] introduced a family of efficient high-performance LLMs trained on carefully curated datasets, enabling strong results with fewer parameters. Concurrently, Anthropic’s Claude models [ 3 ] emphasized constitutional AI and safer alignment techniques for large-scale language systems. More recently, models such as PaLM [ 8 ] and Qwen [ 2 ] have continued to push the boundaries of multilingual reasoning, instruction following, and generalization. Together, these works form a rich body of literature that charts the evolution of LLMs from early transformer models to today’s powerful foundation models used across research and industry. Although large language models have demonstrated remarkable performance across diverse tasks, their increasing scale brings significant challenges that limit their practical deployment. Modern LLMs require substantial computational resources, large memory footprints, and high energy consumption, making them difficult to run on edge devices, real-time systems, or cost-constrained environments. Their inference latency can hinder interactive applications, and frequent retraining or fine-tuning exacerbates the computational burden. These limitations have created a strong need for efficient model compression techniques, such as pruning [ 41 ] , knowledge distillation [ 15 ] , and quantization [ 25 ] , which aim to reduce model size, improve inference speed, and lower resource requirements while maintaining accuracy. Motivated by these issues, this research explores how such optimization strategies can make large language models more scalable, accessible, and practical for real-world use. Quantization in deep learning is an optimization technique that reduces the precision of a model’s numerical values, typically converting them from floating-point numbers to lower-precision integers. The authors in [ 25 , 48 ] have published a comprehensive study on the quantization of LLMs range mapping viz, affine quantization, scale quantization, quantization techniques viz., post-training quantization, quantization-aware training, weight quantization and activation-aware weight quantization. Although post-training quantization techniques improve LLM’s computational efficiency and memory footprint, their hand-crafted quantization settings result in poor performance, particularly in very low-bit quantization. This issue is addressed in omnidirectionally calibrated quantization i.e., OmniQuant technique for LLMs. To address the issue of substantial training resources in quantization aware training, effientQAT method was proposed consisting of Block-wise training of all parameters and end-to-end training of quantization parameters [ 6 ] . Authors in [ 26 ] applied low rank adaptation along with quantization aware training simultaneously which reduces the difference between the full-precision and quantized models and greatly enhances generalization in subsequent challenges. Pruning in deep learning is a technique to reduce the size and complexity of a neural network by removing less important parameters, like weights, neurons, or entire layers. In one approach proposed, each weight matrix is parameterized using its low-rank factorization and adaptively rank-1 components are eliminated during training [ 46 ] . A novel and effective pruning method titled Wanda (pruning by Weights and activations), designed to induce sparsity in pretrained LLMs [ 41 ] . A batched greedy pruning method names SlimGPT for rapid and near-optimal pruning is proposed which enhances the accuracy of head-wise pruning error estimation through grouped Cholesky decomposition [ 27 ] . Authors in [ 1 ] proposed Fluctuation-based Adaptive Structured Pruning which formulated structured importance metrics, adaptively searched the global compressed model, and implemented compensation mechanisms to mitigate performance loss. While individual compression techniques—quantization [ 10 ] , pruning [ 28 ] , and knowledge distillation [ 19 ] , have been extensively studied in isolation, real-world deployment scenarios often require combining multiple techniques to achieve aggressive compression ratios while maintaining acceptable performance [ 49 ] . However, the existing literature provides limited guidance on the optimal ordering of these techniques when applied sequentially on small scale LLMs. Recent surveys [ 45 ] identify this as a critical gap: different orderings may exhibit synergistic or antagonistic interactions, and certain sequences may be infeasible due to technical constraints (e.g., quantization’s incompatibility with gradient-based training). The foundational "Deep Compression" work in [ 17 ] used a pipeline of pruning, quantization, and encoding to drastically reduce the AlexNet model’s size. This same principle of applying techniques in a sequence can be extended to modern small-scale LLMs. This work systematically explores compression technique orderings to identify optimal strategies for practitioners deploying compressed LLMs in resource-constrained environments. This research paper is