TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantization
Haodong Wang, Junjie Liu, Zicong Hong, Qianli Liu, Jian Lin, Song Guo, Xu Chen · 2026
Read TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantization by Haodong Wang, Junjie Liu, Zicong Hong, Qianli Liu, Jian Lin, Song Guo, Xu Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01556.txt.
Introduction
Wang and co-authors learn friendlier subspaces for 4-bit model quantization. Four-bit quantization shrinks memory and latency for LLM inference. Its aggressive precision cut can hurt accuracy badly. Prior methods split each weight matrix into two parts. A low-rank part handles outliers at high precision. A residual part holds the bulk at low precision. Those splits minimize real-valued residual energy, not post-quantization error. TwinQuant learns decomposed subspaces that quantize well instead.
It reshapes both parts with learned transformations. Joint optimization runs over Stiefel and general linear manifolds. Distributions flatten and dynamic-range imbalance falls. A fused kernel pipelines the two-stage computation on chip. Across LLaMA3 and Qwen3 models, accuracy nears FP16. Speed rises up to 1 point 8 times over the FP16 baseline.
Catalog record
| CREATOR | Haodong Wang, Junjie Liu, Zicong Hong, Qianli Liu, Jian Lin, Song Guo, Xu Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01556
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantization Haodong Wang, Junjie Liu, Zicong Hong, Qianli Liu, Jian Lin, Song Guo, Xu Chen Abstract 4-bit quantization reduces the memory footprint and latency of large language model inference, but its aggressive precision reduction can severely degrade accuracy. Prior methods address this by decomposing each weight matrix into two components (e.g., via singular value decomposition) and quantizing them separately, assigning the bulk of values to a low-precision residual component while handling outliers with a high-precision low-rank component. However, such decompositions are designed to minimize the real-valued energy of the residual, rather than the post-quantization error of the residual and low-rank components. We propose TwinQuant, a 4-bit quantization framework that learns quantization-friendly decomposed subspaces and jointly reshapes both the low-rank and residual components. TwinQuant learns component-specific transformations via a joint optimization over the Stiefel and general linear manifolds, flattening their distributions and reducing dynamic-range imbalance. To enable efficient end-to-end execution, we further design a fused dual-component kernel that pipelines the two-stage low-rank computation on-chip and merges both components with a single epilogue, avoiding intermediate global-memory traffic. Across LLaMA3 and Qwen3 models, TwinQuant preserves near-FP16 accuracy and delivers up to 1.8 × 1.8\times end-to-end speedup over an FP16 baseline. 1 Introduction Large language models (LLMs) have become core infrastructure for conversational assistants, medical decision support, and code generation ( 8 ; 25 ; 4 ; 21 ; 22 ; 2 ) , driven largely by aggressive scaling from 7B to 70B and even beyond 600B parameters ( 13 ) . However, this growth makes inference increasingly dominated by large matrix multiplications, amplifying both compute and memory-traffic costs and creating a key deployment bottleneck. To relieve this pressure, modern accelerators are rapidly strengthening native support for low-precision arithmetic (e.g., FP8/FP4 on NVIDIA Blackwell) ( 16 ) . Meanwhile, post-training quantization (PTQ) ( 1 ; 12 ; 30 ) compresses weights and runtime activations into low-bit formats, reducing memory footprint and improving throughput, making quantization a key technique for efficient LLM serving. However, aggressively quantizing both weights and activations to low-bit often leads to large accuracy drops because their heavy-tailed and highly anisotropic statistics couple heterogeneous components within the same tensors, causing the quantization error to be dominated by a small subset of directions and magnitudes ( 23 ) . Early approaches apply scaling and matrix transformations ( 1 ; 14 ; 20 ) to re-distribute weights and activations before quantization, but they mainly re-express tensors in a different coordinate system without removing their intrinsic anisotropy: a few dominant directions still dictate the dynamic range, forcing all groups to share limited 4-bit scales and leaving substantial distortion that redistribution alone cannot eliminate. More recent methods are to move beyond a single space and decompose each weight into two components ( 10 ; 27 ) . Representative singular value decomposition (SVD)-based schemes ( 10 ) , where a truncated SVD can absorb outliers into a small high-precision low-rank branch while quantizing the residual. This is effective when singular values decay quickly, so a tiny rank can preserve most outliers in diffusion models. However, we empirically observe much slower rank decay in the mainstream LLMs (as discussed in section 3 ), so tiny rank cannot absorb enough outliers. This creates a dilemma: increasing rank adds substantial memory and compute overhead, while a small rank leaves a large residual with sharp 4-bit accuracy loss. To break this trade-off, we propose TwinQuant , a dual-branch 4-bit quantization framework for LLMs built on a learnable subspace decomposition that learns how to split each weight into two complementary components under a limited rank budget, with both components being quantization-friendly. By learning the decomposition, TwinQuant reshapes the statistics of each component to be more amenable to low-bit quantizers (e.g., reducing dynamic range and directional skew) so that 4-bit distortion is lowered without increasing the truncation rank or introducing substantial extra compute and memory overhead. A naive strategy is to perform an SVD decomposition and then train the quantization parameters (e.g., quantization scales and clipping thresholds) for the low-rank and residual components. However, this faces two practical issues: (i) tuning these parameters across all layers and both components introduces non-trivial training overhead, often approaching the cost of full quantization-aware training; (ii) the two components are strongly coupled: the residual is defined by what the low-rank reconstruction does not capture, so updating the quantization of one component shifts the effective distribution seen by the other, making joint calibration unstable and difficult to coordinate. Inspired by rotation-based quantization ( 14 ; 1 ) to reshape tensor statistics for low-bit quantization without heavy training overhead, TwinQuant uses parameter-efficient global and layer-specific transforms to adaptively reallocate the subspaces of the low-rank and residual components, directly minimizing quantization error. In addition, these transforms can be seamlessly folded into the weights offline, avoiding additional overhead at inference time. To further reduce the runtime cost of the two-component computation, we design a specialized kernel that jointly executes both components while minimizing memory-access overhead. We summarize our contributions as follows: • We propose TwinQuant , a 4-bit quantization paradigm for LLMs that learns quantization-friendly decomposed subspaces and efficiently reallocates information between the low-rank and residual components. • We introduce a joint optimization scheme that over the Stiefel and general linear manifolds to learn better component-specific transformations for two components to minimize end-to-end quantization error. • We design a fused dual-component kernel that pipelines the two-stage low-rank computation on-chip and merges both components in a single epilogue, reducing kernel launches and avoiding intermediate global memory traffic. • We conduct extensive experiments on mainstream LLMs and system-level evaluations on both workstation and server-class GPUs, demonstrating that TwinQuant preserves near-FP16 accuracy while achieving up to 1.8\times speedup. 2 Background & Related Work Quantization is a widely used technique to accelerate the linear operators in Transformer layers by reducing their arithmetic precision. For a tensor \mathbf{X} , symmetric quantization is defined as \mathbf{Q}_{\mathbf{X}}=\mathrm{round}\!\left(\frac{\mathbf{X}}{s_{\mathbf{X}}}\right),\quad s_{\mathbf{X}}=\frac{\max(|\mathbf{X}|)}{q_{\max}}, (1) where \mathbf{Q}_{\mathbf{X}} is the integer representation of \mathbf{X} , s_{\mathbf{X}} is the scaling factor, and q_{\max} is the maximum representable integer. For signed integers with k bits, q_{\max}=2^{k-1}-1 . The dequantized tensor is then given by \mathcal{Q}(\mathbf{X})=s_{\mathbf{X}}\,\mathbf{Q}_{\mathbf{X}} . For a linear layer with activation \mathbf{X}\in\mathbb{R}^{b\times m} and weight \mathbf{W}\in\mathbb{R}^{m\times n} , \mathbf{X}\mathbf{W}\approx\mathcal{Q}(\mathbf{X})\,\mathcal{Q}(\mathbf{W})=s_{\mathbf{X}}s_{\mathbf{W}}\cdot\mathbf{Q}_{\mathbf{X}}\mathbf{Q}_{\mathbf{W}}. (2) To exploit low-precision compute units on modern GPUs, \mathbf{Q}_{\mathbf{X}} and \mathbf{Q}_{\mathbf{W}} usually use the same bit-width; otherwise, one must be dequantized on the fly to match the other, which undermines quantization speedups. We also define the quantization error as \mathbf{E}_{\mathbf{X}}=\mathbf{X}-\mathcal{Q}(\mathbf{X}) . Per-channel Scaling. The input activations are often rich in outliers. To mitigate their impact on quantization, a popular way is to apply channel-wise scaling over weights and activations \hat{\mathbf{X}}\hat{\mathbf{W}}=\left(\mathbf{X}\,\mathrm{diag}(\lambda)^{-1}\right)\cdot\left(\mathrm{diag}(\lambda)\mathbf{W}\right), where \lambda is the channel-wise scaling factor which jointly calibrates the magnitudes of input activations and weights and can be folded into the model weights offline, eliminating extra runtime computation. SmoothQuant ( 24 ) selects the scaling factors via grid search to balance the activation–weight trade-off, while FlatQuant ( 20 ) learns the scaling factors directly as trainable parameters. Low-Rank Decomposition. Recent works show that low-rank decomposition can effectively reduce the errors introduced by low-bit quantization in diffusion models ( 10 ; 27 ) . SVDQuant ( 10 )