BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference

Wonsuk Jang, Thierry Tambe · 2025

Read BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference by Wonsuk Jang, Thierry Tambe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-01144.txt.

Description

Wonsuk Jang and Thierry Tambe address the cost of running large language models in a 2025 paper. Bigger models need more memory and power, so engineers shrink the numbers inside them. Fine-grained shrinking helps, but one format rarely fits all blocks of data well. Outlying values make the fit worse.

Their method gives each block of data its own best number format, chosen from a small set of variants. A two-stage step handles new data as the model runs, while staying cheap in energy. Tests on two well-known models showed clear accuracy gains over a standard format, using fewer bits and staying close to full precision.

Catalog record

CREATORWonsuk Jang, Thierry Tambe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-01144

Browse the full library · About the Tower of Babel