ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning
Baohao Liao Christian Herold Seyyed Hadi Hashemi, Stefan Vasilev Shahram Khadivi Christof Monz · 2025
Read ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning by Baohao Liao Christian Herold Seyyed Hadi Hashemi, Stefan Vasilev Shahram Khadivi Christof Monz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13089.txt.
Introduction
Baohao Liao and co-authors present ClusComp in a 2025 paper, a way to shrink large language models and then fine tune the shrunk version. Quantization cuts model size by storing weights in fewer bits, but quality drops sharply at very low bit widths. Worse, standard fine tuning does not work on quantized models, and past workarounds fall short of full fine tuning.
ClusComp groups weight matrices into codebooks and fine tunes them block by block. It beat other methods at two to four bit widths in the team's tests. It also pushed compression to a single bit while using little fine tuning. Its fine tuning beat past quantization based approaches and rivaled full precision fine tuning. As a headline result, it can compress and fine tune a very large model on one high memory graphics card.
Catalog record
| CREATOR | Baohao Liao Christian Herold Seyyed Hadi Hashemi, Stefan Vasilev Shahram Khadivi Christof Monz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13089
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning Baohao Liao Christian Herold Seyyed Hadi Hashemi, Stefan Vasilev Shahram Khadivi Christof Monz Abstract As large language models (LLMs) scale, model compression is crucial for edge deployment and accessibility. Weight-only quantization reduces model size but suffers from performance degradation at lower bit widths. Moreover, standard finetuning is incompatible with quantized models, and alternative methods often fall short of full finetuning. In this paper, we propose ClusComp , a simple yet effective compression paradigm that clusters weight matrices into codebooks and finetunes them block-by-block. ClusComp (1) achieves superior performance in 2-4 bit quantization, (2) pushes compression to 1-bit while outperforming ultra-low-bit methods with minimal finetuning, and (3) enables efficient finetuning, even surpassing existing quantization-based approaches and rivaling full FP16 finetuning. Notably, ClusComp supports compression and finetuning of 70B LLMs on a single A6000-48GB GPU. Figure 1: Left: Compression quality of ClusComp. Only the most competitive baselines are shown here. Please refer to Table A.1 and A.2 for a full comparison. Methods in triangle use more calibration samples than the ones in circle. Right: Compression efficiency of ClusComp that consists of sequential clustering and reconstruction steps. E.g., compressing a 70B LLM takes 25.6h + 2GB for clustering, and 3.5h + 26GB for reconstruction on 1 GPU. 1 Introduction Large language models (LLMs) have garnered significant acclaim and success across various domains and applications ( Dubey et al., 2024 ; Brown et al., 2020 ; Raffel et al., 2020a ) . With ongoing advancements, the scope and complexity of released LLMs have witnessed exponential growth, with some LLMs encompassing > 50B parameters ( DeepSeek-AI et al., 2024 ; Zhang et al., 2022 ; Scao et al., 2022 ) . This remarkable upscaling introduces considerable challenges, particularly when deploying these models or granting their accessibility to users with constrained resources. To address these challenges, weight-only post-training quantization (PTQ) has emerged as an effective approach. PTQ methods can generally be classified into three categories: statistic-based, gradient-based, and codebook-based approaches. Statistic-based methods ( Dettmers et al., 2024 ; Lin et al., 2024 ; Frantar et al., 2022 ) determine the quantization grid based on the weight value distribution, whereas gradient-based methods ( Shao et al., 2024 ; Ma et al., 2024 ) optimize the quantization grid with some calibration samples. Codebook-based methods ( Egiazarian et al., 2024 ; van Baalen et al., 2024 ; Kim et al., 2024 ; Park et al., 2024 ) cluster similar weight elements to the shared quantized centroids, employing non-uniform quantization and pushing the limits to extremely low bit levels. However, these methods continue to struggle with low-bit quantization and the presence of outliers in weights, leading to significant performance degradation. Another challenge PTQs encounter is their limited support for finetuning, which is crucial for adapting LLMs to various downstream tasks. Finetuning LLMs is computationally expensive due to their large scale and the need to cache activations and store optimizer states. PTQ, which compresses LLMs, appears to be a promising approach for finetuning as it reduces the memory requirements for loading LLMs. However, most quantization techniques use a round-to-nearest operation, which does not support gradient backpropagation. Typically, parameter-efficient methods ( Dettmers et al., 2023 ; Li et al., 2024c ; Liao et al., 2024 ) are employed to train the added parameters while keeping the quantized LLMs frozen, bypassing this limitation. However, this fine-tuning approach presents two major drawbacks: (1) freezing quantized LLMs prevents further reduction of quantization errors during finetuning; (2) The low-rank nature of most parameter-efficient methods restricts their expressiveness ( Biderman et al., 2024 ; Liao and Monz, 2024 ) . In this paper, we propose a simple while effective paradigm that mainly applies Clus tering to Comp ress LLMs, referred to as ClusComp . Additionally, ClusComp can function as a parameter and memory-efficient finetuning method. Our preliminary experiments reveal that LLMs are increasingly difficult to quantize, mainly due to the growing frequency of outliers in their weight matrices (§ 3 ). Based on this observation, we propose using clustering instead of quantization to compress LLMs, retaining all values in FP16 format to circumvent issues arising from outlier quantization (§ 3.1 ). To further reduce compression errors, we minimize block-wise output discrepancies between the compressed and uncompressed blocks, using a limited set of calibration samples (§ 3.3 ). In addition, by incorporating an inexpensive end-to-end recovery finetuning step, we can even push the compression rate to the 1-bit level. Since all parameters remain in FP16 after compression, ClusComp fully supports standard neural training, thus allowing the seamless finetuning of compressed LLMs on various downstream tasks (§ 3.4 ). We begin by evaluating the effectiveness of ClusComp in the context of model compression across 2 language modeling tasks and 6 zero-shot reasoning tasks. ClusComp consistently surpasses various baselines at 2-4 levels, even achieving a perplexity of < 13 at the 2-bit level on WikiText2 ( Merity et al., 2017 ) for all LLMs (§ 4.1 ). Following recovery finetuning, ClusComp’s performance at 2-bit and 1-bit levels approaches that of the FP16 model, with an accuracy of 57.8 vs 68.6 for the 2-bit Llama-3-8B and 51.4 vs 75.4 for the 1-bit Llama-3-70B (§ 4.2 ). Additionally, ClusComp demonstrates its utility as a parameter-efficient ( <1\% ) and memory-efficient (42GB for Llama-3-70B) finetuning method, outperforming quantization-based and memory-efficient finetuning approaches, while matching the performance of full finetuning (§ 4.3 ). Figure 2: The Llama series becomes progressively harder to quantize. Left & Middle: From Llama-2 to Llama-3, all methods struggle more with lower-bit quantization, though ClusComp - is the least affected. Right: From Llama-1 to Llama-3, the average weight kurtosis across most layers increases. 2 Related Works Quantization refers to the process of converting floating-point values into discrete levels, thereby reducing the bit-width required and minimizing memory consumption during model loading. Taking the symmetric uniform quantization as an example, a weight matrix \bm{W} is quantized as follows: \displaystyle\bm{W}_{q}=\text{clamp}(\lfloor\frac{\bm{W}}{s}\rceil,-2^{b-1},2^{b-1}-1) where the scale factor s=\frac{\text{max}(|\bm{W}_{\text{min}}|,|\bm{W}_{\text{max}}|)}{2^{b}-1} , b denotes the bit-width, and \lfloor\rceil represents the round-to-nearest (RTN) operation. Since the quantization grid is uniform, its effectiveness is contingent on the distribution of the weight values. In cases where the weight matrix contains a significant number of outliers or is quantized to lower bit-widths, the resulting quantization error may be substantial. Post-training quantization (PTQ) methods, such as GPTQ ( Frantar et al., 2022 ) , AWQ ( Lin et al., 2024 ) , and OmniQuant ( Shao et al., 2024 ) , apply quantization to a model after training with minimal computational resources. However, these approaches, which rely on uniform quantization, are significantly impacted by the presence of outliers in the weight matrices. Recent methods ( Dettmers et al., 2024 ; Yuan et al., 2024 ; Huang et al., 2024a ) address this challenge by retaining salient weights in FP16 format, thereby maintaining strong performance at lower bit widths. Nonetheless, these mixed-precision quantization techniques require specially optimized CUDA kernels to either enhance or preserve inference speed. Closely related to our proposed method, ClusComp, are works such as GPTVQ ( van Baalen et al., 2024 ) , QuIP# ( Tseng et al., 2024 ) and SqueezeLLM ( Kim et al., 2024 ) which implement quantized codebooks