Intrinsic Structure as a Proxy for Saliency: SVD-Based Weight Preservation for Mixed-Precision Quantization in Large Language Models

Shashank Landge 1 , Abhishek Patil 2 , Tejas kamble 3 , Bhushan Buddhivant 4 and Priyanka Joshi 5 · 2025

Read Intrinsic Structure as a Proxy for Saliency: SVD-Based Weight Preservation for Mixed-Precision Quantization in Large Language Models by Shashank Landge 1 , Abhishek Patil 2 , Tejas kamble 3 , Bhushan Buddhivant 4 and Priyanka Joshi 5 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01343.txt.

Introduction

Shashank Landge and co-authors tackle the size of large language models. Big models are hard to run on cheap hardware. One fix shrinks each weight to fewer bits. If all weights shrink the same way, key weights get hurt and skill drops.

Their method keeps the most key weights at higher care. It uses a math split of each weight block to judge which parts matter most. Parts with strong inner form get more bits. The rest get fewer. This mixed plan aims to keep skill high while the model stays small enough for plain machines.

Catalog record

CREATORShashank Landge 1 , Abhishek Patil 2 , Tejas kamble 3 , Bhushan Buddhivant 4 and Priyanka Joshi 5
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01343

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Intrinsic Structure as a Proxy for Saliency: SVD-Based Weight Preservation for Mixed-Precision Quantization in Large Language Models Shashank Landge 1 , Abhishek Patil 2 , Tejas kamble 3 , Bhushan Buddhivant 4 and Priyanka Joshi 5 Abstract As Large Language Models (LLMs) continue to scale in parameter count, deploying them on commodity hardware has become increasingly challenging. Post-Training Quantization (PTQ) addresses this by reducing the precision of model weights, typically to 4-bit or lower. However, uniform quantization often leads to significant performance degradation due to the presence of “outlier features”—weights that, while few in number, are critical for maintaining model accuracy. Current state-of-the-art methods such as AWQ (Activation-aware Weight Quantization) and SpQR (Sparse Quantization Representations) rely on calibration data to identify these salient weights via activation magnitudes or Hessian sensitivity. In scenarios where data privacy is paramount or calibration data is unavailable, these methods are inapplicable. In this work, we propose a data-free, structure-aware hypothesis: that the weights identified as Principal Components via Singular Value Decomposition (SVD) are intrinsically important to the model’s downstream performance. We introduce a novel selection heuristic that preserves the top- k k weights aligned with the principal components in FP32, while aggressively quantizing the residual weights. We compare our method against activation-aware (AWQ) and second-order (SpQR) methods across GLUE benchmarks (MRPC, RTE, QNLI) using a DistilBERT backbone. Our experiments reveal that structural importance is highly correlated with functional importance. On the challenging RTE task, our SVD-based method achieves an accuracy of 66.06%, outperforming both AWQ (65.34%) and SpQR (65.34%) at high protection budgets, validating that intrinsic matrix structure can serve as a robust proxy for weight saliency without the need for forward passes or calibration data. , 5 [email protected] I Introduction The recent explosion in the capabilities of Large Language Models (LLMs), such as GPT-4 and Llama-2, has been driven largely by scaling laws that necessitate massive parameter counts. While effective, these sizes present a barrier to entry for deployment on edge devices or consumer-grade GPUs. Reducing the memory footprint of these models without retraining has become a critical area of research. Quantization, the process of mapping high-precision floating-point weights (e.g., FP16) to lower-precision integers (e.g., INT4), offers a promising solution. However, aggressive quantization often results in a collapse of model performance. Research has shown that this degradation is not uniform; rather, it is driven by a small subset of “salient” weights or “outlier features” that are extraordinarily sensitive to quantization noise [ 5 ] . I-A The Challenge of Saliency Detection The core challenge in mixed-precision quantization is efficiently identifying which weights to preserve in high precision (the “salient” set) and which to compress. Existing state-of-the-art methods rely on data-driven metrics: • AWQ [ 1 ] : Observes activation magnitudes on a calibration set, hypothesizing that weights processing large activations are more important. • GPTQ [ 4 ] / SpQR [ 2 ] : Utilize second-order derivatives (Hessian matrix) computed from calibration data to determine which weights minimize reconstruction error. While effective, these methods introduce a dependency on calibration data. If the calibration set distribution differs from the deployment distribution, or if privacy constraints prevent access to data entirely, these methods cannot be applied. I-B Our Contribution We propose a shift from data-aware importance to structure-aware importance. Drawing inspiration from Parameter-Efficient Fine-Tuning (PEFT) methods like PiSSA [ 3 ] , which utilize Singular Value Decomposition (SVD) to identify trainable parameters, we hypothesize that the principal components of a weight matrix encode its primary linguistic capabilities. Hypothesis: The weights identified as Principal Components by SVD factorization are intrinsically important for the specific downstream task. Therefore, if we preserve those k Principal weights in FP32 and quantize the rest of the original weights, the resulting model will retain high accuracy. Our specific contributions are: 1. We formalize an SVD-based heuristic for mixed-precision quantization that requires zero calibration data. 2. We conduct a rigorous comparative analysis (“The Battle”) against AWQ and SpQR on three GLUE tasks: MRPC, RTE, and QNLI. 3. We demonstrate that our method is competitive with data-aware methods, and in specific low-resource regimes (like the RTE task), actually outperforms them. 4. We analyze the Intersection over Union (IoU) of selected indices, proving that SVD intrinsically identifies 60-70% of the same weights that complex Hessian-based methods identify. II Related Work II-A Post-Training Quantization (PTQ) PTQ aims to compress models without the massive cost of retraining. Early methods like LLM.int8() [ 5 ] utilized vector-wise quantization to handle outliers. SmoothQuant [ 7 ] proposed mathematically smoothing activation outliers into weights to make quantization easier. These methods focus on making the entire model quantifiable, whereas our work focuses on selective preservation. II-B Data-Aware Selection AWQ (Activation-aware Weight Quantization) [ 1 ] shifted the paradigm by noting that preserving 1% of salient weights can fix quantization error. Their heuristic, based on activation magnitude |W|X , is fast but requires forward passes. SpQR (Sparse Quantization Representations) [ 2 ] takes this further by identifying sensitive weights via the Hessian inverse (Optimal Brain Surgeon framework [ 8 ] ). While theoretically optimal, SpQR is computationally expensive ( O(d^{3}) ) and data-dependent. II-C Low-Rank Decomposition Low-Rank Adaptation (LoRA) [ 6 ] demonstrated that weight updates during fine-tuning have a low intrinsic rank. PiSSA [ 3 ] recently proposed initializing the LoRA adapters using the principal singular values of the original weight matrix. Our work adapts the PiSSA initialization logic for the distinct problem of quantization sensitivity, essentially asking: “Are the weights that are good for initialization also the weights that are bad to quantize?” III Methodology We consider a pre-trained weight matrix W\in\mathbb{R}^{d_{out}\times d_{in}} . Our goal is to decompose W into a sparse high-precision component S (Salient) and a dense low-precision component Q (Quantized), such that: W\approx S+Q (1) where S retains FP32 precision but has high sparsity (only k non-zero elements), and Q is fully quantized to 4-bit. III-A Selection Heuristics We compare three distinct heuristics for populating the set S . III-A 1 Baseline: Random Selection As a lower bound, we select k indices uniformly at random: I_{rand}\sim\text{Uniform}(1,\dots,|W|) (2) III-A 2 Data-Aware: AWQ We pass a calibration batch X through the network. The importance score for weight w_{ij} is: \text{Score}_{AWQ}(w_{ij})=|w_{ij}|\cdot||X_{j}||_{2} (3) where X_{j} is the column of activations corresponding to input channel j . This prioritizes weights that act on large activation features. III-A 3 Second-Order: SpQR (Hessian) We compute the empirical Hessian H=\frac{2}{N}X^{T}X . The saliency is derived from the Optimal Brain Damage (OBD) objective: \text{Score}_{SpQR}(w_{ij})=\frac{w_{ij}^{2}}{[H^{-1}]_{jj}} (4) Due to the computational cost of inverting H , we use a damped inverse with \lambda=0.01 . This identifies weights which, if perturbed, cause the maximum increase in L2 loss. III-A 4 Proposed: SVD-Based Selection (Our Method) We perform Singular Value Decomposition on the weight matrix W : W=U\Sigma V^{T} (5) We define the “Principal Structure” W_{pri} as the reconstruction using only the top r singular values (we use r=8 following PiSSA literature): W_{pri}=U_{:,:r}\cdot\text{diag}(\Sigma_{:r})\cdot V_{:,:r}^{T} (6) Our selection score is simply the magnitude of this principal reconstruction: \text{Score}_{SVD}(w_{ij})=|(W_{pri})_{ij}| (7) The top- k indices with the highest scores are selected. Note that this method relies solely on the weights W and requires no input data X . III-B Quantization Mechanism For

Read the complete text