Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization
Ian W. Kennedy, Nafise Sadat Moosavi · 2026
Read Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization by Ian W. Kennedy, Nafise Sadat Moosavi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08118.txt.
Introduction
Kennedy and Moosavi show codebook initialization dominates extreme language-model quantization with additive quantization. This coding enables extreme shrinking with constant-time lookup for edge use. Yet at 2-bit precision it often fails badly, even with long search and tuning. Greedy stepwise initialization often lands the model in poor regions that beam search and PV-tuning struggle to escape. The team studies this behavior through the size ratio rho, linking weight groups to codebook size, and offers OA-EM, an output-aware EM start with Hessian-weighted Mahalanobis distance.
Catalog record
| CREATOR | Ian W. Kennedy, Nafise Sadat Moosavi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43778 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08118
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization Thanks: Preprint. Under review. Ian W. Kennedy, Nafise Sadat Moosavi Abstract Additive quantization enables extreme LLM compression with O ( 1 ) O(1) lookup-table dequantization, making it attractive for edge deployment. Yet at 2-bit precision, it often fails catastrophically, even with extensive search and fine-tuning. We show that the dominant bottleneck is codebook initialisation. Greedy sequential initialisation frequently places the model in poor optimisation regions that subsequent beam search and PV-tuning struggle to overcome. We analyse this behaviour through the representational ratio ρ = N / K M \rho=N/K^{M} , which characterises the relationship between weight groups and codebook capacity, and propose OA-EM, an output-aware EM initialisation method using Hessian-weighted Mahalanobis distance. Across compression rates, search budgets, and three architectures (Llama 3.2 3B, Llama 3.1 8B, Qwen 2.5 3B), OA-EM consistently produces better solutions after PV-tuning and dominates the quality–compute frontier. The severity of the bottleneck scales with ρ \rho : moderate at 3 bpp but extreme at 2 bpp, where poor initialisation can degrade perplexity by orders of magnitude. More broadly, our results highlight the importance of optimisation geometry in compressed model spaces, where initialisation can dominate subsequent search and fine-tuning. Our code is available at 1 1 1 https://github.com/kenno94-IK/aqlm-oaem . 1 Introduction Large language model deployment on consumer hardware requires aggressive weight compression ( Gholami et al. 2021 ) . While 4-bit quantization is near-lossless ( Frantar et al. 2023 ; Lin et al. 2024 ) , the 2-bit regime remains challenging: each parameter is encoded with only 4 possible values, leaving almost no room for approximation error. This regime is particularly relevant for the 3B–8B parameter range, where 2-bit compression enables deployment on single consumer GPUs and mobile edge devices: hardware where memory, not compute, is the binding constraint. Two paradigms compete in this regime. Structured methods—such as lattice codebooks ( Tseng et al. 2024a ) , trellis codes ( Tseng et al. 2024b ) , and grouped lattice VQ ( Zhang et al. 2025 ) —achieve strong perplexity via mathematically constrained codebook geometry, but require active computation (e.g. Babai rounding, matrix–vector multiplication) during inference. In contrast, free-form additive methods ( Egiazarian et al. 2024 ) learn unconstrained codebooks, enabling O(1) lookup-table (LUT) dequantization with zero multiply-accumulate (MAC) operations per weight group—a pure memory read that is critical for edge deployment on ARM CPUs, microcontrollers, and mobile SoCs where ALU cycles, not memory bandwidth, are the primary bottleneck ( Egiazarian et al. 2024 ) . Our work focuses on this free-form family. Additive quantization ( Egiazarian et al. 2024 ) encodes each group of g weights as a sum of M codewords, drawn from learned codebooks of K{=}256 entries each. When performance degrades at extreme compression, the common response is to increase computational effort: wider beam search, more epochs per layer, or larger calibration sets. We show that, in this regime, such strategies target the wrong bottleneck. The dominant factor is initialisation: search performed within a poorly initialised region of the solution space yields limited improvement. The key insight is a regime transition governed by the ratio of weight groups to representational capacity: Definition 1.1 (Representational Ratio) . For M additive codebooks with K entries each and N weight groups per layer, the representational ratio is \rho=N/K^{M} . When \rho<1 (overcomplete), there are more representable points than weight groups and initialisation errors can often be absorbed. When \rho>1 (undercomplete), weight groups compete for limited codebook capacity, and initial placement becomes critical. For Llama 3.2 3B: Rate M K^{M} \rho Regime 3 bpp 3 16.8\text{M} {\approx}\,0.07 Overcomplete 2 bpp 2 65\text{K} {\approx}\,18 Undercomplete This 256\times capacity reduction is not a gradual degradation—it is a qualitative change in behaviour. At 3 bpp, greedy initialisation costs 0.65 perplexity points. At 2 bpp, it leads to severe degradation on Llama 3.2 3B (WikiText-2 perplexity 352.39 at beam 4, 60.61 at beam 8 vs. 7.28 FP16), and even quadrupling the beam width to 16 reduces this only to 46.01. To address this, we propose OA-EM (Output-Aware Expectation-Maximisation), which refines each codebook’s initialisation via iterative EM ( Dempster et al. 1977 ) using Hessian-weighted Mahalanobis distance derived from calibration activations, directly minimising output reconstruction error rather than weight-space distance. Beyond the algorithm itself, our primary contribution is the empirical and theoretical observation that initialisation strongly influences the optimisation trajectory of compressed models . Whenever learned codebooks are initialised through greedy sequential fitting, the representational ratio \rho predicts when initialisation becomes critical. We demonstrate this through three lines of evidence: Basin persistence. OA-EM’s advantage persists after end-to-end PV-tuning ( Malinovskii et al. 2024 ) across beam widths, epoch budgets, compression rates, model scales, and architectures. On Llama 3.2 3B, PV-tuning compresses a 43-point perplexity gap to 0.23, yet OA-EM remains better in every configuration. Asymmetric search scaling. Increasing beam width from 8 to 16 improves OA-EM (11.53 \rightarrow 11.49 post-PV) but worsens the greedy baseline (11.76 \rightarrow 12.01 post-PV), suggesting that additional search is beneficial primarily when initialisation is already good. Pareto dominance. OA-EM dominates the quality–compute frontier, achieving lower perplexity and better downstream accuracy at matched compute budgets. Contributions. 1. We introduce the representational ratio \rho and show that it predicts when quantization becomes sensitive to initialisation (§ 3.3 ). 2. We propose OA-EM—output-aware EM with Hessian-weighted Mahalanobis distance—as a drop-in replacement for greedy initialisation (§ 3.4 ). 3. Through systematic post-PV analysis across beam widths, epoch budgets, compression rates (2 bpp and 3 bpp), and three models (Llama 3.2 3B, Llama 3.1 8B, Qwen 2.5 3B), we show that initialisation can lead to persistent optimisation differences whose severity scales with \rho (§ 5.2 ). 2 Related Work Scalar PTQ. A large body of work studies scalar post-training quantization (PTQ) for LLM compression, where each weight is assigned an independent low-bit representation. GPTQ ( Frantar et al. 2023 ) quantizes layer-wise using approximate second-order information; AWQ ( Lin et al. 2024 ) protects salient weights via activation-based scaling. SmoothQuant ( Xiao et al. 2023 ) migrates quantization difficulty from activations to weights via per-channel scaling, and OmniQuant ( Shao et al. 2024 ) learns weight clipping and equivalence transformations end-to-end. Such scalar methods degrade rapidly below 3 bpp because scalar codes cannot efficiently use the information budget at extreme compression. LLM.int8() ( Dettmers et al. 2022 ) demonstrated mixed-precision decomposition for 8-bit quantization. SpQR ( Dettmers et al. 2024 ) and SqueezeLLM ( Kim et al. 2024 ) extend sensitivity-aware principles to 3–4 bits. QLoRA ( Dettmers et al. 2023 ) combines 4-bit quantization with low-rank adaptation for efficient fine-tuning, demonstrating the practical demand for aggressive compression on consumer hardware. These scalar approaches share our motivation that weight sensitivity should guide compression, but operate in the regime where the initialisation problem we study does not arise. Structured vector quantization for LLMs. Beyond scalar quantization, several works explore vector quantization with structured codebooks for extreme compression. QuIP ( Chee et al. 2023 ) enabled 2-bit LLMs via incoherence processing. QuIP# ( Tseng et al. 2024a ) introduced the randomised Hadamard transform and E8 lattice codebooks, because the codebook structure is fixed (not learned), QuIP# avoids the initialisation problem we study. SpinQuant (