KLLM : Fast LLM Inference with K-Means Quantization

Xueying Wu, Baijun Zhou, Zhihui Gao, Yuzhe Fu, Qilin Zheng, Yintao He † , and Hai “Helen” Li · 2025

Read KLLM : Fast LLM Inference with K-Means Quantization by Xueying Wu, Baijun Zhou, Zhihui Gao, Yuzhe Fu, Qilin Zheng, Yintao He † , and Hai “Helen” Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23035.txt.

Introduction

Xueying Wu and co-authors present OASIS, a 2025 arXiv paper on speeding up large language models. The catalog lists the work under a different title, but the text itself names and describes OASIS. Big models eat memory and power when they answer, and shrinking their numbers to save space brings a trade-off. Plain shrinking either costs time to undo or blurs the model's answers.

OASIS uses precomputed lookup tables to multiply shrunken weights and signals without undoing the shrink first. Its tables run far smaller and far more in parallel than older table methods. A side step spots rare outlying values and fixes their errors on the fly. Tests show a small drop in accuracy, about two percent, with a threefold speed gain over a rival chip design.

Catalog record

CREATORXueying Wu, Baijun Zhou, Zhihui Gao, Yuzhe Fu, Qilin Zheng, Yintao He † , and Hai “Helen” Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23035

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

KLLM : Fast LLM Inference with K-Means Quantization Xueying Wu, Baijun Zhou, Zhihui Gao, Yuzhe Fu, Qilin Zheng, Yintao He † , and Hai “Helen” Li Abstract Large language model (LLM) inference poses significant challenges due to its intensive memory and computation demands. Weight and activation quantization (WAQ) offers a promising solution by reducing both memory footprint and arithmetic complexity. Traditional WAQ designs rely on uniform integer quantization for hardware efficiency, but often suffer from significant model performance degradation at low precision. In contrast, K-Means quantization, a non-uniform technique, achieves higher accuracy by aligning with the Gaussian-like distributions of weights and activations in LLMs. However, two key challenges prevent the efficient deployment of K-Means-based WAQ designs for LLM inference: (1) The non-uniform structure of K-Means-quantized data precludes direct execution on low-precision compute units, necessitating dequantization and floating-point matrix multiplications (MatMuls) during inference. (2) Activation outliers hinder effective low-precision quantization. Offline thresholding methods for outlier detection degrade model performance substantially, while existing online detection techniques introduce significant runtime overhead. To address the aforementioned challenges and fully unleash the potential of K-Means-based WAQ for LLM inference, in this paper, we propose KLLM , an LLM inference accelerator for efficient execution with K-Means-quantized weights and activations. KLLM features an index-based computation scheme for efficient execution of MatMuls and nonlinear operations on K-Means-quantized data, which avoids most of the dequantization and full-precision computations. Moreover, KLLM incorporates a lightweight outlier detection engine, Orizuru , that efficiently identifies the top- k k largest and smallest elements in the activation data stream during online inference. Extensive experiments show that, KLLM achieves speedups of 8.66 × \times , 2.12 × \times and energy efficiency improvements of 214.76 × \times , 2.79 × \times compared to the A100 GPU and SpAtten, respectively. Department of Electrical and Computer Engineering, Duke University Email: {xueying.wu, baijun.zhou, zhihui.gao, yuzhe.fu, qilin.zheng, yintao.he, hai.li}@duke.edu I Introduction Large language models (LLMs) have achieved impressive advancements, demonstrating versatile applications such as chatbots [ 1 , 18 ] , code generation [ 41 , 46 ] , and circuit design [ 20 , 14 ] . However, the enormous sizes of LLMs pose significant challenges for inference, demanding substantial memory [ 27 , 5 ] and computational resources [ 7 , 12 ] . Matrix multiplications (MatMuls) dominate the execution time of LLM inference [ 58 ] . To address this bottleneck, weight and activation quantization (WAQ) methods have been proposed to enable low-precision MatMuls, which reduce both memory usage and computational costs [ 61 , 3 , 55 ] . Existing WAQ methods primarily leverage integer quantization, which represents data with equally spaced quantization centroids [ 55 , 23 ] . However, this uniform spacing fails to align with the actual distributions of LLM weights and activations, which often exhibit non-uniform Gaussian-like patterns, with most values concentrated around the mean [ 22 , 25 ] . Therefore, integer WAQ (I-WAQ) methods often suffer from high quantization noises and significant degradation in model performance, especially at low quantization precisions. For example, as shown in Figure 1 (a), the W4A4 perplexity (PPL) of LLaMA-2-7B [ 48 ] on WikiText-2 [ 34 ] with SmoothQuant [ 55 ] is 702.3, which is 127 \times higher than the FP16 model. To mitigate PPL degradation, recent I-WAQ methods such as QuaRot [ 3 ] and Atom [ 61 ] employ techniques of Hadamard transformations and group-wise quantization to reduce quantization noise. Unfortunately, as shown in Figure 1 (a), these techniques still result in noticeable PPL degradation compared to FP16 models. Moreover, as shown in Figure 1 (b), these techniques also lead to increased runtime overhead, which is caused by additional matrix operations and reduced computational parallelism of MatMuls. Overall, existing I-WAQ methods exhibit noticeable model performance degradation and high runtime overhead, despite leveraging integer compute units. Fig. 1: (a) PPL comparison across W4A4 I-WAQ and K-WAQ methods. (b) MatMul runtime and (c) runtime breakdown of W4A4 kernels on A100 GPU vs. KLLM accelerator. Runtime in both (b) and (c) are measured for a MatMul between a 4096\times 4096 weight matrix and a 1\times 4096 activation vector. Compared to uniform integer quantization, the non-uniform quantization method of K-Means clustering [ 33 ] could better align with data distributions and reduces quantization noises. This is because K-Means determines quantization centroids through an optimization process that allocates more centroids to regions with higher data density. This non-uniform allocation improves data representation fidelity, especially in low-precision configurations [ 22 , 25 ] . However, the non-uniform nature of K-Means-quantized data makes it incompatible with integer compute units. Consequently, data must be dequantized back to FP16 format, and MatMuls are performed on FP16 compute units. As illustrated in Figure 1 (b), this leads to even higher runtime for K-Means WAQ (K-WAQ) compared to I-WAQ methods when implemented on an NVIDIA A100 GPU [ 36 ] . Figure 1 (c) further reveals that codebook lookups during dequantization dominate the MatMul runtime, accounting for 63% of the total execution time. Thus, to fully unleash the potential of K-WAQ for LLM inference acceleration, it is crucial to design an efficient computation scheme and customized compute units capable of directly processing the integer indices of K-Means-quantized data, eliminating the need for dequantization. To address the aforementioned challenges, in this work, we propose KLLM , an LLM inference accelerator for efficient execution with K-Means-quantized weights and activations. KLLM introduces an index-based computation scheme, along with customized low-precision compute units that directly process the integer indices of K-Means-quantized data. As illustrated in Figure 1 (c), during MatMul computation, KLLM significantly reduces the runtime of codebook lookups and FP16 multiply-accumulates (MACs), while introducing minimal overhead for index-based operations. KLLM achieves a 7.93 \times speedup for a W4A4 MatMul with K-WAQ compared to the A100 GPU. This computation scheme also provides significant speedup for nonlinear operations in LLMs. Moreover, we design an efficient top- k largest and smallest element detection engine named Orizuru for dynamically identifying activation outliers, which effectively mitigates quantization errors. We compare KLLM with state-of-the-art quantization algorithms and accelerators. For the algorithmic performance, KLLM achieves a PPL of 5.90 for LLaMA-2-7B [ 48 ] model on WikiText-2 [ 34 ] dataset with W4A4 K-WAQ, which corresponds to a PPL degradation of 0.43 compared to the FP16 model. This PPL degradation is 34% lower than that of Atom [ 61 ] . For the hardware performance, KLLM under W4A4 configuration achieves average speedups of 8.66 \times , 2.12 \times and energy efficiency improvements of 214.76 \times , 2.79 \times compared to the A100 GPU and SpAtten [ 49 ] , respectively. The main contributions of this work are summarized as: • We propose an efficient index-based computation scheme that supports both MatMul and nonlinear operations for LLMs with K-Means-quantized data. • We design KLLM , an LLM inference accelerator that executes the proposed index-based scheme on customized low-precision compute units with high efficiency. • We propose Orizuru , an efficient top- k largest and smallest element detection engine for dynamic outlier identification. II Motivation As discussed in § I , I-WAQ designs face significant limitations in low-precision LLM quantization, which include: (1) noticeable PPL degradation compared to FP16 models, and (2) substantial runtime

Read the complete text