MatGPTQ: Accurate and Efficient Post-Training Matryoshka Quantization

Maximilian Kleinegger, Elvir Crnčević, Dan Alistarh · 2026

Read MatGPTQ: Accurate and Efficient Post-Training Matryoshka Quantization by Maximilian Kleinegger, Elvir Crnčević, Dan Alistarh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03537.txt.

Description

One stored model that can serve at several precisions would simplify deploying language models on varied hardware. Maximilian Kleinegger and 2 co-authors deliver that with MatGPTQ in a 2026 paper. In nested quantization, a low-precision model is read out of a high-precision one by using fewer bits per weight. Building such models usually needs expensive training, though.

MatGPTQ reframes the job as multi-precision error compensation, done in a single pass using one small calibration set. It also refines the storage format so an r-bit model reads exactly r bits. And it ships the first dedicated inference kernels, integrated into the vLLM serving system. The pipeline beats earlier nested methods and reaches speedups up to three and a half times in the low-bit regime.

Catalog record

CREATORMaximilian Kleinegger, Elvir Crnčević, Dan Alistarh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03537

Browse the full library · About the Tower of Babel