Fast NF4 Dequantization Kernels for Large Language Model Inference

Xiangbo Qi, Chaoyi Jiang, Murali Annavaram · 2026

Read Fast NF4 Dequantization Kernels for Large Language Model Inference by Xiangbo Qi, Chaoyi Jiang, Murali Annavaram (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02556.txt.

Description

Qi, Jiang and Annavaram speed up NF4 dequantization, the step that expands 4-bit NormalFloat weights back to 16-bit form during large language model inference. Their lightweight shared-memory optimization exploits the large latency gap between shared and global memory while using only 64 bytes of shared memory per thread block. It runs 2.0 to 2.2 times faster than the BitsAndBytes kernels across three models, with up to 1.54 times faster end-to-end inference.

Catalog record

CREATORXiangbo Qi, Chaoyi Jiang, Murali Annavaram
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28520
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.02556

Browse the full library · About the Tower of Babel