Fast NF4 Dequantization Kernels for Large Language Model Inference
Xiangbo Qi, Chaoyi Jiang, Murali Annavaram · 2026
Read Fast NF4 Dequantization Kernels for Large Language Model Inference by Xiangbo Qi, Chaoyi Jiang, Murali Annavaram (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02556.txt.
Description
Qi, Jiang and Annavaram speed up NF4 dequantization, the step that expands 4-bit NormalFloat weights back to 16-bit form during large language model inference. Their lightweight shared-memory optimization exploits the large latency gap between shared and global memory while using only 64 bytes of shared memory per thread block. It runs 2.0 to 2.2 times faster than the BitsAndBytes kernels across three models, with up to 1.54 times faster end-to-end inference.
Catalog record
| CREATOR | Xiangbo Qi, Chaoyi Jiang, Murali Annavaram |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28520 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.02556