MatGPTQ: Accurate and Efficient Post-Training Matryoshka Quantization
Maximilian Kleinegger, Elvir Crnčević, Dan Alistarh · 2026
Read MatGPTQ: Accurate and Efficient Post-Training Matryoshka Quantization by Maximilian Kleinegger, Elvir Crnčević, Dan Alistarh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03537.txt.
Description
One stored model that can serve at several precisions would simplify deploying language models on varied hardware. Maximilian Kleinegger and 2 co-authors deliver that with MatGPTQ in a 2026 paper. In nested quantization, a low-precision model is read out of a high-precision one by using fewer bits per weight. Building such models usually needs expensive training, though.
MatGPTQ reframes the job as multi-precision error compensation, done in a single pass using one small calibration set. It also refines the storage format so an r-bit model reads exactly r bits. And it ships the first dedicated inference kernels, integrated into the vLLM serving system. The pipeline beats earlier nested methods and reaches speedups up to three and a half times in the low-bit regime.
Catalog record
| CREATOR | Maximilian Kleinegger, Elvir Crnčević, Dan Alistarh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-03537