When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon

Mohamed Amine Bergach · 2026

Read When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon by Mohamed Amine Bergach (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05699.txt.

Description

Shrinking the memory cache of a language model is normally a bargain: less memory, slightly worse answers, maybe more speed. Mohamed Amine Bergach shows the bargain inverts on Apple Silicon. His int4 cache, a single fused Metal kernel exposed as a HuggingFace cache class, outruns the full-precision cache on two small models.

The kernel combines a randomized Fourier transform, per-channel scaling, and nibble packing. Memory use fell threefold. On Gemma-3 1B it ran three to eight percent faster per token across prefix lengths, with perplexity essentially unchanged on short prompts. Unified memory bandwidth, not arithmetic, is the bottleneck it exploits.

Catalog record

CREATORMohamed Amine Bergach
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05699

Browse the full library · About the Tower of Babel