Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Pingzhi Tang, Ruijie Zhou, Fanxu Meng, Wenjie Pei, Muhan Zhang · 2026
Read Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation by Pingzhi Tang, Ruijie Zhou, Fanxu Meng, Wenjie Pei, Muhan Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22716.txt.
Introduction
Pingzhi Tang and four co-authors break the block grid that rules how large models are quantized and tuned. Quantization stores weights in fewer bits to save space and speed. Their 2026 preprint notes that block-wise scaling, the norm, trades away flexibility for ease.
They show element-wise scaling can match block speed while expressing more. Their trick models the scale field as a smooth, low-rank surface. One method then serves both quantization and adaptation. Finer scales mean less quality lost per bit saved. The work loosens a deep habit of the field.
Catalog record
| CREATOR | Pingzhi Tang, Ruijie Zhou, Fanxu Meng, Wenjie Pei, Muhan Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22716
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation Pingzhi Tang, Ruijie Zhou, Fanxu Meng, Wenjie Pei, Muhan Zhang Abstract Current quantization methods for LLMs predominantly rely on block-wise structures to maintain efficiency, often at the cost of representational flexibility. In this work, we demonstrate that element-wise quantization can be made as efficient as block-wise scaling while providing strictly superior expressive power by modeling the scaling manifold as continuous low-rank matrices ( S = B A S=BA ). We propose Low-Rank Decomposed Scaling (LoRDS), a unified framework that rethinks quantization granularity through this low-rank decomposition. By “breaking the blocks” of spatial constraints, LoRDS establishes a seamless efficiency lifecycle: it provides high-fidelity PTQ initialization refined via iterative optimization, enables joint QAT of weights and scaling factors, and facilitates high-rank multiplicative PEFT adaptation. Unlike additive PEFT approaches such as QLoRA, LoRDS enables high-rank weight updates within a low-rank budget while incurring no additional inference overhead. Supported by highly optimized Triton kernels, LoRDS consistently outperforms state-of-the-art baselines across various model families in both quantization and downstream fine-tuning tasks. Notably, on Llama3-8B, our method achieves up to a 27.0% accuracy improvement at 3 bits over NormalFloat quantization and delivers a 1.5 × \times inference speedup on NVIDIA RTX 4090 while enhancing PEFT performance by 9.6% on downstream tasks over 4bit QLoRA, offering a robust and integrated solution for unified compression and adaptation of LLMs. † † affiliationnotice: Equal contribution 1 Introduction (a) QLoRA (b) LoRDS Figure 1 : Architectural comparison between (a) QLoRA and (b) our proposed LoRDS. (a) QLoRA employs standard block-wise quantization where parameters within each block share a static scaling factor. During fine-tuning, it introduces an auxiliary, full-precision additive adapter ( W+BA ), which cannot be merged into the quantized weights, leading to additional inference latency. (b) LoRDS fundamentally rethinks the scaling mechanism by decomposing the block-wise scaling matrix S into a low-rank product BA . This multiplicative formulation ( W\odot BA ) serves a dual purpose: it can be refined via PTQ or QAT to achieve more granular scaling, or directly fine-tuned for PEFT. LoRDS enables high-rank parameter updates and incurs zero additional inference overhead, as the learned scaling factors are naturally absorbed into the dequantization process. Large Language Models (LLMs, 7 ; 39 ) have demonstrated remarkable capabilities across a wide range of natural language processing tasks, fundamentally reshaping the landscape of artificial intelligence. However, their exceptional capabilities come at significant resource demands: the immense parameter counts impose large memory footprints and computational costs, making both efficient deployment and downstream adaptation significant challenges. To mitigate these bottlenecks, extensive research has been dedicated to model compression and efficient tuning. Model Quantization ( 37 ; 44 ; 3 ; 10 ; 4 ; 9 ) reduces memory usage by approximating high-precision weights with low-bit numbers, significantly lowering storage requirements. Parameter-Efficient Fine-Tuning (PEFT, 41 ; 18 ) methods, such as LoRA ( 19 ) , freeze the pretrained model and update only a small set of auxiliary parameters, thereby reducing training costs. Efforts have also been made to integrate these two paradigms. Methods like QLoRA ( 12 ) and LoftQ ( 27 ) combine quantization with LoRA adapters, PEQA ( 23 ) directly finetunes the quantization parameters. Despite these advancements, current approaches face several critical limitations. First, the dominant quantization paradigm relies heavily on block-wise quantization ( 11 ) , where weights are grouped into blocks that share quantization parameters. While this approach implicitly assumes that neighboring parameters possess similar statistical properties, this assumption lacks theoretical guarantees and often fails to capture weight distributions, especially in the presence of outliers. Second, the landscape of efficient LLM utilization remains fragmented: Post-Training Quantization (PTQ, 16 ; 28 ), Quantization-Aware Training (QAT, 17 ; 22 ; 24 ; 30 ; 15 ; 6 ), and Quantized PEFT are typically treated as distinct optimization problems with separate methodologies, lacking a unified framework. Finally, existing quantized PEFT methods like QLoRA introduce additive adapters alongside quantized weights. Unlike full-precision fine-tuning where adapters can be merged, the mixed-precision nature of QLoRA prevents lossless merging, incurring additional latency and computational overhead during inference. To address these challenges, we propose Low-Rank Decomposed Scaling (LoRDS), a unified framework that rethinks the granularity of quantization scaling. LoRDS improves upon traditional block-wise quantization by breaking the rigid block structure. Instead of independent scalars for each block, we model the scaling factors as a continuous low-rank matrix decomposed via Singular Value Decomposition (SVD). This approach effectively occupies a representational space between coarse per-block scaling and dense per-element scaling ; while LoRDS can be seamlessly initialized from standard block-wise statistics, its continuous low-rank formulation offers strictly superior expressive power. This formulation allows for global coherence and more flexible representations of the weight magnitude distribution. Crucially, LoRDS unifies the lifecycle of LLM efficiency: (1) As a PTQ method, it can be initialized from standard block-wise settings and refined through low-cost alternating optimization, achieving significantly lower quantization error. (2) As a QAT method, the full-precision weights and the decomposed scaling matrices ( A and B ) can be trained jointly using Straight-Through Estimators (STE), pushing the limits of performance. (3) As a PEFT method, by fine-tuning only the low-rank scaling matrices A and B , LoRDS achieves superior performance compared to additive methods like QLoRA and LoftQ. Crucially, unlike LoRA-based methods that are restricted to low-rank additive updates, the multiplicative nature of LoRDS enables effectively high-rank parameter updates. Furthermore, as the adaptation is intrinsic to the dequantization scaling process, LoRDS does not incur additional inference overhead compared to standard quantized models. We extensively evaluate LoRDS across various model families (e.g., LLaMA-3, Qwen-3) and downstream tasks. Experimental results demonstrate that LoRDS consistently minimizes quantization errors in both PTQ and QAT settings by leveraging its continuous low-rank decomposed scaling mechanism. Furthermore, thanks to its ability to facilitate high-rank weight updates, LoRDS achieves superior adaptation performance in PEFT tasks while maintaining a smaller floating-point parameter budget than existing baselines. Crucially, our approach eliminates the inference bottleneck inherent in additive adapters, delivering up to a 1.5 \times inference speedup compared to QLoRA on NVIDIA RTX 4090 GPUs. In summary, our main contributions are as follows: • We propose Low-Rank Decomposed Scaling (LoRDS), a novel quantization framework that interpolates between coarse per-block and dense per-element scaling with learnable low-rank matrices, effectively capturing complex weight distributions. • We present a unified formulation that seamlessly covers PTQ, QAT, and PEFT, offering a versatile solution for the entire LLM efficiency lifecycle. • We achieve state-of-the-art performance in both quantization reconstruction and downstream fine-tuning tasks, outperforming existing additive PEFT methods without introducing any inference latency. • We implement highly optimized Triton kernels for inference of LoRDS, bridging the gap between theoretical efficiency and real-world deployment speed. 2 Related Work Post-Training Quantization (PTQ) is widely adopted due to its ability to be directly applied to pretrained LLMs, typically with the aid of a small calibration dataset. PTQ methods are commonly divided into weight-only quantization and weight-activation quantization. Weight-only quantization focuses on compressing model parameters into low-bit representations. GPTQ ( 16 ) leverages second-order information for accurate layer-wise quantization. AWQ ( 28 ) identifies salient weight channels based on activation distributions and protects them