Pushing the Envelope of LLM Inference on AI-PC

Evangelos Georganas, Dhiraj Kalamkar, Alexander Heinecke · 2025

Read Pushing the Envelope of LLM Inference on AI-PC by Evangelos Georganas, Dhiraj Kalamkar, Alexander Heinecke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06753.txt.

Introduction

Evangelos Georganas and co-authors squeeze language models onto personal computers in this 2025 paper. New models quantized to about two bits per weight nearly match full precision quality while slashing memory and energy. The software that runs them, though, had not kept pace.

Working from the bottom up, the team wrote two bit microkernels tuned for modern central processors, reaching close to the hardware's roofline speed. Wired into an inference pipeline, their two bit models beat the leading runtime by more than two times and ran up to seven times faster than sixteen bit inference. They extended the kernels to graphics processors with mixed precision and into a popular serving framework, gaining similar speedups there as well.

Catalog record

CREATOREvangelos Georganas, Dhiraj Kalamkar, Alexander Heinecke
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06753

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pushing the Envelope of LLM Inference on AI-PC Evangelos Georganas, Dhiraj Kalamkar, Alexander Heinecke Abstract The advent of ultra-low-bit LLM models (1/1.58/2-bit), which match the perplexity and end-task performance of their full-precision counterparts using the same model size, is ushering in a new era of LLM inference for resource-constrained environments such as edge devices and AI PCs. While these quantization advances promise models that are more cost-effective in terms of latency, memory, throughput, and energy consumption, the computational efficiency of state-of-the-art (SOTA) inference runtimes (e.g., bitnet.cpp) used to deploy them remains underexplored. In this work, we take a bottom-up approach: we first design and implement 1-bit and 2-bit microkernels optimized for modern CPUs, achieving peak computational efficiency across a variety of CPU platforms. We integrate these microkernels into a state-of-the-art LLM inference framework, namely PyTorch-TPP, and present end-to-end inference results with 2-bit models that outperform the current SOTA runtime bitnet.cpp by up to 2.2 × \times , and deliver up to 7 × \times speedup compared to the 16-bit model inference. Our optimized runtime advances the state of LLM inference on AI PCs and edge devices, paving the way for efficient deployment of ultra-low-bit LLM models. I Introduction Recent advances in Quantization-Aware Training (QAT) LLM methods have produced ultra-low bit (1-bit, 1.58-bit and 2-bit) quantized models that match the perplexity and end-task performance of their full-precision counterparts using the same model size [ 1 , 2 ] . Despite these advancements that yield ultra-low bit LLM models with the potential to be cost-effective in terms of latency, memory, throughput, and energy consumption, the available runtimes capable of serving inference with such models are limited. For example, the state-of-the-art inference runtime bitnet.cpp is specifically designed for ternary LLMs, with a relatively narrow range of applicable model architectures [ 3 ] . Additionally, although the bitnet.cpp runtime improves upon alternatives (e.g. llama.cpp) being up to 6 \times faster by introducing a specialized mixed-precision matrix multiplication (mpGEMM) library to facilitate sub-2-bits model inference, its efficiency is not evaluated in terms of optimality or roofline performance. Our preliminary performance analysis showed that 2-bit inference with bitnet.cpp was even slower that state-of-the-art 4-bit inference on CPUs (e.g. see in Figures 9 , 8 , 10 the orange and the magenta bars), highlighting that the bitnet.cpp runtime is far from optimal. In this work we adopt a bottom-up approach to address the performance shortcomings of bitnet.cpp. The key observation is that client/edge inference typically involves single-batch use-cases, which in turn result in memory-bound matrix multiplication invocations (more precisely matrix-vector multiplication) during the execution. By exploiting the reduced bit-width of the model’s weight tensors during inference we can accelerate the corresponding matrix-vector multiplications (e.g. compared to the case with 16-bit weight tensors), since the volume of data read on the critical path is proportional to the bit-width of the weight datatype. First, we designed and implemented 1-bit and 2-bit mixed-precision matrix multiplication (GEMM) routines that up-convert the low-precision weight matrices (1- or 2-bit entries) to 8-bit integers, and perform the fused-multiply-add (FMA) operations using hardware-accelerated instructions on modern CPUs. We introduce novel weight layouts that facilitate the up-convert process, and within the GEMM microkernels we deploy instruction sequences that maximize the throughput of the “up-convert and compute” steps. We also construct performance models for the introduced kernels to assess their efficacy and show that they achieve close to peak roofline performance on modern CPU platforms. Furthermore, we compose these microkernels into optimized multi-threaded GEMM routines that leverage dynamic task scheduling to fully exploit the heterogeneous architectures with performance and efficiency cores. We benchmark these multi-threaded GEMM routines on a variety of modern x86 CPU platforms. Finally, we integrate these ultra-low bit GEMM routines in a SOTA inference framework, namely PyTorch TPP [ 4 ] , and showcase end-to-end inference results using 2-bit and 1-bit models. Our optimized 2-bit inference is up to 7 \times faster than the 16-bit inference, and outperforms bitnet.cpp by up to 2.2 \times . The contributions of this work are: • We designed and implemented 1-bit and 2-bit GEMM kernels tailored for CPUs with AVX2 ISA. These kernels leverage novel blocked and interleaved weight tensor layouts to facilitate the up-convert process during inference. We devised performance models that characterize the effectiveness and limitations of these kernels. • We composed these new microkernels into multi-threaded GEMM routines targeting hybrid-core architectures. We benchmarked the new routines on a set of matrices and platforms, showing that they can achieve the attainable memory bandwidth, effectively accelerating the bandwidth-bound matrix-vector multiplication operations proportionally to the bit-width of the weight’s datatypes. • We integrated these ultra-low bit GEMM kernels in our SOTA end-to-end inference pipeline, assessed its performance on a range of models and platforms, and observed speedups up to 7 \times over the 16-bit inference. • We benchmarked our runtime against bitnet.cpp and show that our proposed solution is up to 2.2 \times faster. Finally, we compare our CPU performance with the optimized GPU 2-bit inference within bitnet.cpp, which includes kernels specifically tailored for the A100 platform. Our performance on contemporary x86 AI-PC CPUs is within 2.3 \times -3 \times of the A100 GPU, which has 17 \times -20 \times more bandwidth than the CPU platforms. Thus, our work pushed the envelope of LLM inference on AI-PCs and demonstrates that ultra-low-bit inference on CPUs can approach GPU-level performance. II Related Work II-A LLM Inference and ultra-low bit Quantization In recent years, several methods have been developed by academia and industry to reduce LLM inference costs [ 5 ] , minimize latency and increase efficiency, including sparsification/pruning [ 6 ] , knowledge distillation [ 7 , 8 ] and quantization [ 9 , 10 , 11 ] . Among these, quantization reduces the precision of model weights (e.g. from 16-bits to 4-bits/2-bits/1-bit), thereby lowering memory requirements and potentially accelerating inference, especially when the target hardware supports specialized instructions or when the pipeline operates in a bandwidth-bound regime [ 12 ] . Quantization methods for LLMs generally fall into two categories: Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ). QAT incorporates reduced-precision weights directly into the model’s pretraining or fine-tuning process. By accounting for quantization effects during training, QAT enables the model to adjust its parameters with awareness of limited precision, resulting in better accuracy during quantized inference. In contrast, PTQ applies quantization to a pretrained model without additional training or fine-tuning. Although QAT typically achieves higher accuracy, it is more computationally intensive and requires access to representative training datasets [ 2 ] . As LLMs continue to scale in size and training data, increasing attention has been drawn to scaling laws that balance model size and dataset size to optimize performance and computational efficiency [ 13 , 14 , 15 ] . In parallel, the field is moving towards ultra-low-bit LLMs, motivated by the significant gains in memory and compute efficiency [ 16 , 1 , 2 ] . This trend calls for a re-evaluation of scaling laws to consider the impact of quantization on model performance. Recent work shows that 1.58-bit (ternary) and 2-bit quantization deliver performance comparable to full-precision models in the size–accuracy trade-off, and generally outperform both 4-bit and binary quantization [ 2 ] . II-B Accelerating

Read the complete text