MatGPTQ: Accurate and Efficient Post-Training Matryoshka Quantization Maximilian Kleinegger, Elvir Crnčević, Dan Alistarh Abstract Matryoshka Quantization (MatQuant) is a recent quantization approach showing that a single integer-quantized model can be served across multiple precisions, by slicing the most significant bits (MSB) at inference time. This enables a single checkpoint to cover a wide range of memory and latency budgets, but renders quantization much more challenging. In particular, the initial MatQuant relies on expensive quantization-aware training (QAT) variants, rather than fast one–shot post training quantization (PTQ), and lacks open-source and kernel support. We address all of these limitations by introducing Post-Training Matryoshka Quantization (MatGPTQ), a new PTQ pipeline that produces a single parent model jointly optimized for multiple target precisions in one-shot, based on a small calibration set. MatGPTQ casts Matryoshka quantization as a multi–precision objective with bit-slicing and cross–bit error compensation, resulting in an algorithm that produces a multi-bit-width, “sliceable” model in a single pass. We also incorporate a new budget–aware search for heterogeneous per–layer bit-witdhs and provide efficient kernels that implement slicing and mixed–precision execution. Across standard LLMs and benchmarks, MatGPTQ preserves high–bit accuracy while substantially improving performance at low-bit-witdh settings. Overall, we establish a new state of the art for Matryoshka–style post–training quantization and make single–checkpoint, multi–precision deployment open and practical. Code is available at https://github.com/IST-DASLab/MatGPTQ . † † affiliationnotice: ∗ Work done during an internship at ISTA, 1 Introduction Quantization is currently the most practical way to shrink the memory and bandwidth costs of large language models (LLMs): by mapping full-precision weights to low-bit representations, one can enable single‑GPU or edge deployment, while mostly retaining model accuracy. One‑shot, post‑training quantization (PTQ) is attractive for its low cost and strong accuracy in the 3-4 bit regime ( Frantar et al. 2022 ; Kurtic et al. 2024 ) , whereas optimization‑based methods such as QAT and block‑wise, gradient‑calibrated PTQ (e.g., OmniQuant ( Shao et al. 2023 ) ) achieve higher accuracy, but at the expense of extra computational cost. The recent Matryoshka Quantization (MatQuant) work of Nair et al. 2025 provided a fresh approach to model compression by turning integer bit representations into a nested representation : a single “parent” model quantized at the highest precision can be sliced (via most-significant bit (MSB) extraction) at runtime to yield lower‑bit models, enabling sliced variants of a single parent checkpoint to serve many deployment budgets. The MatQuant paper demonstrates single‑model, multi‑precision training and shows that slicing supports bit‑width interpolation, e.g., 6‑ and 3‑bit from an 8‑bit parent. Further, the paper suggested the possibility of Mix-and-Match trade‑offs between cost and accuracy, by non-uniform per-layer slicing. Follow-up work such as Any‑Precision LLM ( Park et al. 2024 ) explored an orthogonal upscaling route for multi-scale PTQ, by starting from a low‑bit model, and progressively adding bits for increasing accuracy. Despite this promise, the initial MatQuant evaluations are confined to finetuning‑based quantization methods, such as QAT and block-wise QAT (OmniQuant), as opposed to the cheaper and more popular one‑shot PTQ methods, such as GPTQ ( Frantar et al. 2022 ; Shao et al. 2023 ; Nair et al. 2025 ) . MatQuant’s description fixes the trained target set to the canonical triplet of {8,4,2} bit-widths (with interpolation to {6,3}), rather than offering a general multi‑target PTQ recipe and implementation for arbitrary bit-widths ( Nair et al. 2025 ) . While heterogeneous quantization is discussed, the paper does not explore how to search for bit-widths in order to balance accuracy and deployment costs. Further, no code nor kernels were released at the time of writing. Contributions. We address these gaps with Post‑Training Matryoshka Quantization (MatGPTQ), a PTQ framework that optimizes a single parent model jointly for a user‑specified set of bit‑widths R (e.g., \{2,3,4,6,8\} bits per parameter). Algorithmically, we (i) cast Matryoshka training as a new type of multi‑precision GPTQ objective with cross‑bit error sharing and MSB slicing; (ii) provide open and efficient compression and GPU kernel implementations supporting more than two nested precisions, including integer 2–8 bits with bit‑width interpolation; and (iii) integrate a principled evolutionary search for heterogeneous per‑layer bit‑width assignment under fixed memory budgets ( Sieberling et al. 2025 ) . We release kernels, packing, and a vLLM ( Kwon et al. 2023 ) integration required for end-to-end deployment. Experimental results. We evaluate MatGPTQ across a suite of modern LLMs, including LLaMA 3.1 8B, Qwen3 (8B/14B), and Phi-3-Medium. Our results demonstrate that MatGPTQ successfully produces a single nested model that performs similarly or better relative to independently-trained, non-nested GPTQ baselines. Crucially, MatGPTQ preserves accuracy at higher bit-widths (4-8 bits, within 0.7% of baseline) while achieving gains in the challenging low-bit regime, showing a 1.34% average improvement at 3 bits, relative to the uniform quantization baseline. Overall, the method matches or outperforms the QAT approach of Nair et al. 2025 , and also exhibits strong “interpolation” capabilities, by maintaining accuracy when slicing models at intermediate bit-widths (e.g., 6-bit) for which the model did not optimize explicitly. Furthermore, we validate the integration of evolutionary search for heterogeneous, non-uniform per-layer quantization. This “Mix-and-Match” approach yields configurations that are often Pareto-superior to standard uniform PTQ, achieving comparable or higher accuracy at lower average bit-witdhs, even down to 2.5 bits. Finally, we benchmark our optimized CUDA kernels, confirming significant end-to-end latency improvements. For instance, 3-bit inference on LLaMA 3.1 8B achieves nearly a 3x end-to-end speedup compared to FP16 execution on NVIDIA GPUs, demonstrating the practical viability of deploying sliceable models. All in all, MatGPTQ shows that efficient, and open-source post-training Matryoshka quantization is eminently viable, lowering the barrier of entry for multi-precision deployment. 2 Background and Related Work Quantization is often applied as a post-training (PTQ) step, in one-shot to already trained models. We focus on such one-shot PTQ methods in this paper, by contrast to more expensive quantization-aware training (QAT). For PTQ, one can differentiate between data-free approaches, which use simpler round-to-nearest (RTN) projections ( Dettmers 2016 ; Badri & Shaji 2023 ) , but tend to drop accuracy even at 4-bits per parameter, and data-aware approaches such as GPTQ ( Frantar et al. 2022 ) or AWQ ( Lin et al. 2024 ) , which utilize a calibration set that is passed through the model during quantization. Specifically, a common approach in PTQ is to quantize each linear layer \ell with weights W_{\ell} , in linear order, to produce the quantized weights Q_{\ell} . Please see Section 3 for an overview of the GPTQ algorithm. Matryoshka Quantization (MatQuant) ( Nair et al. 2025 ) is a general-purpose multi-scale quantization technique whose objective is to generate “nested” models, in the sense that the higher-bitwdith models “contain” the lower-bit-witdh models. More precisely, in MatQuant, for certain bit-widths r , one can slice the most significant r bits out of a c -bit model, with 0