Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression

Hengyi Zhu, Zhendong Mi, Grace Li Zhang, Shaoyi Huang · 2026

Read Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression by Hengyi Zhu, Zhendong Mi, Grace Li Zhang, Shaoyi Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08568.txt.

Introduction

Hengyi Zhu, Zhendong Mi, Grace Li Zhang, and Shaoyi Huang shrink large language models by choosing compression levels per prompt. Standard singular value methods cut every input at the same fixed rank, yet the best rank shifts from prompt to prompt and depends heavily on the calibration data. Their framework, PARSE, trains a small router to pick ranks ahead of time.

Similar prompts share rank patterns that stay stable while answers generate, so chosen ranks can be served from a cache for speed. Added to four existing compression methods, PARSE lifts task accuracy and speeds both prompt reading and answer generation. Model deployment gains quality and speed from treating each prompt on its own terms.

Catalog record

CREATORHengyi Zhu, Zhendong Mi, Grace Li Zhang, Shaoyi Huang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08568

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression Hengyi Zhu, Zhendong Mi, Grace Li Zhang, Shaoyi Huang Abstract Large language models (LLMs) have rapidly grown in scale, creating substantial memory and computational costs that hinder efficient deployment. Singular value decomposition (SVD) has emerged as an effective post-training compression technique, but existing SVD-based methods rely on static rank truncation, applying a fixed prefix of singular components to all inputs regardless of their diversity. We identify two limitations of this static design: the optimal rank varies across individual prompts, and the selected rank is sensitive to the choice of calibration set, leading to suboptimal performance across diverse inputs. To address these challenges, we propose PARSE , a post-training framework for P rompt- A ware R ank S election as E xperts in SVD-compressed LLMs. PARSE trains a linear router offline to perform prompt-aware rank selection, decoupling it from calibration information by supervising the router against dense-model outputs on a large-scale corpus. We further observe that rank-selection patterns are shared across semantically similar prompts and remain stable across decoding steps, allowing appropriate rank subsets to be served directly from a pattern cache at inference. Complemented by expert memory aggregation and kernel fusion for system-level efficiency, PARSE is orthogonal to existing SVD-based pipelines and consistently improves both model quality and inference efficiency. Integrated with four representative SVD-based methods, PARSE improves average task accuracy by up to 10% at a compression ratio of 0.6 on LLaMA-7B, and achieves up to 2.5 × \times prefill and 2.4 × \times decode speedup over native SVD execution. 1 Introduction Large language models (LLMs) have become foundational across diverse applications, from natural language generation and translation to complex mathematical and code reasoning [ 22 , 26 ] . Guided by scaling laws [ 32 ] , modern LLMs such as GPT [ 6 ] , PaLM [ 9 ] , LLaMA [ 50 ] , DeepSeek [ 22 ] , and Qwen [ 59 ] have grown to hundreds of billions of parameters. While such scale unlocks remarkable capabilities [ 48 , 13 ] , it also incurs heavy computational overhead that hinders inference on resource-constrained hardware and inflates the economic and environmental costs of large-scale serving [ 65 , 53 , 15 , 45 ] . Improving LLM efficiency has therefore become a critical challenge for real-world deployment [ 52 , 51 ] . Common compression strategies, including quantization [ 18 , 37 , 28 ] , pruning [ 39 , 17 , 49 ] , and distillation [ 21 , 60 ] , often depend on specific hardware [ 12 , 37 ] or require costly retraining [ 4 , 8 ] . In contrast, low-rank compression decomposes each weight matrix into three matrices and approximates it by discarding small singular values, directly reducing both parameter count and computational cost. Recent advances in low-rank compression for LLMs, including ASVD [ 62 ] , SVD-LLM [ 55 ] , Basis Sharing [ 51 ] , and AdaSVD [ 36 ] , have demonstrated that low-rank approximations can shrink model size with limited performance degradation. Nevertheless, existing approaches still face two notable challenges. (1) Input-agnostic rank allocation. Existing SVD-based methods typically determine the truncation rank through a one-shot calibration on a fixed dataset, yielding a single static rank applied uniformly to all inputs at inference time. This rigid allocation overlooks the fact that the rank required to faithfully preserve model behavior is itself input prompt-dependent, varying with the complexity and semantics of each prompt. A globally fixed rank is therefore inevitably suboptimal for some inputs and leads to unstable performance across heterogeneous prompts. (2) Calibration-domain mismatch. The static rank obtained from one-shot calibration is tightly coupled to the calibration set. When downstream tasks deviate semantically from this set, the chosen rank becomes misaligned with the target distribution, resulting in degraded performance on downstream inputs. To address these challenges, we propose PARSE , a post-training framework for P rompt- A ware R ank S election as E xperts in SVD-compressed LLMs, orthogonal to existing SVD-based compression baselines. PARSE trains a linear router offline to learn prompt-aware rank selection and serves the selected rank subsets from a rank cache at inference. Specifically, we reformulate each SVD-compressed weight matrix as a mixture of rank experts, allowing the router to select a prompt-specific subset from a discrete set of independent rank components. To decouple rank selection from calibration information, the router is supervised against dense-model outputs on a large-scale corpus. To eliminate router overhead at inference, we retrieve cached subsets for semantically similar prompts and reuse the selected subset across decoding steps within the same prompt. We further introduce system-level optimizations, including expert memory aggregation and kernel fusion, to reduce runtime overhead. Our contributions are summarized as follows: • We identify two limitations of static rank truncation in SVD-compressed LLMs: (1) rank selection is sensitive to both individual input prompts within the same dataset and the choice of calibration dataset, leading to suboptimal performance. • We propose PARSE , a post-training framework that trains a linear router offline to perform prompt-aware rank selection, decoupling rank selection from calibration information by supervising the router against dense-model outputs on a large-scale corpus. PARSE is orthogonal to existing SVD-based compression baselines and consistently improves their performance. • We observe that rank patterns are shared across semantically similar prompts and remain stable across decoding steps within the same prompt, enabling rank subset retrieval at prefilling stage and rank reuse at decoding stage to reduce inference overhead. We further develop expert memory aggregation and kernel fusion to reduce system-level overhead. 2 Preliminaries For a weight matrix W\in\mathbb{R}^{m\times n} , its SVD is given by W=U\Sigma V^{\top} , where U\in\mathbb{R}^{m\times m} and V\in\mathbb{R}^{n\times n} are orthogonal matrices, and \Sigma\in\mathbb{R}^{m\times n} is a diagonal matrix with singular values in descending order. Letting r_{\max}=\min(m,n) , this decomposes W into a sum of rank components: W=\sum_{i=1}^{r_{\max}}\sigma_{i}u_{i}v_{i}^{\top}, (1) where \sigma_{i} , u_{i} , and v_{i} are the i -th singular value and left and right singular vectors. By the Eckart–Young–Mirsky theorem [ 14 ] , retaining the top- r components \tilde{W}=\sum_{i=1}^{r}\sigma_{i}u_{i}v_{i}^{\top} minimizes the weight reconstruction error \|\tilde{W}-W\|_{F}^{2} among all rank- r approximations. Minimizing weight reconstruction error does not, however, directly minimize activation reconstruction loss. The compression objective relevant to inference is: \|\tilde{W}X-WX\|_{F}^{2}, (2) where X\in\mathbb{R}^{n\times d} denotes the layer input activations over d tokens. To establish a direct correspondence between singular values and this loss, SVD-LLM [ 55 ] introduces a data whitening step: a whitening matrix S\in\mathbb{R}^{n\times n} is derived via Cholesky decomposition [ 42 ] of XX^{\top} , enforcing S^{-1}XX^{\top}(S^{-1})^{\top}=I . SVD is then applied to WS instead of W , under which the activation reconstruction loss from truncating the i -th component equals exactly \sigma_{i} . The compressed weight is recovered as W^{\prime}=U_{r}\Sigma_{r}V_{r}^{\top}S^{-1} , and this pipeline has been widely adopted in subsequent work [ 52 , 27 ] . In practice, X is estimated from a static calibration dataset, and the rank- r approximation is applied uniformly to all inputs at inference time. 3 Observations Following SVD-LLM [ 55 ] and subsequent works [ 54 , 52 , 51 , 27 ] , existing SVD-based

Read the complete text