FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast
Wenhao Wu 1,∗, Zishan Shao 1,2,∗,†, Kangning Cui 4, Jinhee Kim 1, Yixiao Wang 2,3, Hancheng Ye 3, Danyang Zhuo 3, Yiran Chen 2 1 Department of Electrical & Computer Engineering, Duke University 2 Department of Statistical Science, Duke University 3 Department of Computer Science, Duke University 4 Department of Computer Science, Wake Forest University · 2026
Read FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast by Wenhao Wu 1,∗, Zishan Shao 1,2,∗,†, Kangning Cui 4, Jinhee Kim 1, Yixiao Wang 2,3, Hancheng Ye 3, Danyang Zhuo 3, Yiran Chen 2 1 Department of Electrical & Computer Engineering, Duke University 2 Department of Statistical Science, Duke University 3 Department of Computer Science, Duke University 4 Department of Computer Science, Wake Forest University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08314.txt.
Introduction
Wenhao Wu, Zishan Shao, Kangning Cui, Jinhee Kim, and coauthors make low-rank transformers serve faster in practice. Cutting a model down with singular value decomposition shrinks its size on paper. In production, the factorized weights fragment execution. The overhead also differs between the prompt phase and token-by-token decoding.
FlashSVD maps several public compression families into one factorized format. It serves them with phase-specific kernels. It adds dense key-value decoding and packed MLP execution. It also replays per-layer CUDA graphs. In serving tests, decoding ran up to about two and a half times faster. Average gains were solid across families. Compression pays only when the runtime is rebuilt around it.
Catalog record
| CREATOR | Wenhao Wu 1,∗, Zishan Shao 1,2,∗,†, Kangning Cui 4, Jinhee Kim 1, Yixiao Wang 2,3, Hancheng Ye 3, Danyang Zhuo 3, Yiran Chen 2 1 Department of Electrical & Computer Engineering, Duke University 2 Department of Statistical Science, Duke University 3 Department of Computer Science, Duke University 4 Department of Computer Science, Wake Forest University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08314
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast Wenhao Wu 1,∗, Zishan Shao 1,2,∗,†, Kangning Cui 4, Jinhee Kim 1, Yixiao Wang 2,3, Hancheng Ye 3, Danyang Zhuo 3, Yiran Chen 2 1 Department of Electrical & Computer Engineering, Duke University 2 Department of Statistical Science, Duke University 3 Department of Computer Science, Duke University 4 Department of Computer Science, Wake Forest University Abstract SVD-based Low-rank compression reduces transformer parameters and nominal FLOPs, but these savings often translate poorly into real LLM serving speedups. We show that this gap is largely a runtime problem: factorized checkpoints fragment execution paths, and the resulting overhead differs substantially between prefill and autoregressive decode. We present FlashSVD v1.5 , a unified inference runtime for serving SVD-compressed transformers. FlashSVD v1.5 maps diverse public SVD compression families to a common factorized representation and combines phase-specific kernels with dense-KV decode, packed MLP execution, and per-layer CUDA-graph replay to reorganize the low-rank serving path into a thin runtime. Across representative decoder-serving settings, FlashSVD v1.5 achieves up to 2.55 × 2.55\times decode and 2.39 × 2.39\times end-to-end speedup, and it attains 1.48 × 1.48\times average decode and 1.44 × 1.44\times average end-to-end speedup across multiple popular SVD compression families. These results suggest that practical low-rank acceleration requires runtime co-design, not compression algorithms alone. Our code is available at: https://github.com/Zishan-Shao/FlashSVD . 1 Introduction Modern Large Language Models (LLMs) achieve strong performance across diverse tasks, but their rapidly increasing size makes deployment increasingly difficult. Singular value decomposition (SVD) has recently emerged as a prominent direction for LLM compression [ 31 , 26 , 10 , 11 , 13 , 8 , 16 , 25 , 23 , 15 , 17 ] . These approaches reduce model size and nominal FLOPs while remaining hardware-friendly. In practice, however, these theoretical savings often do not translate into comparable inference speedups. This gap arises from a mismatch between low-rank checkpoints and serving runtimes. First, fragmented checkpoint formats, such as SVD-LLM v1/v2 [ 26 , 25 ] and Basis Sharing [ 22 ] , require separate runtime handling. Second, factorizing a large weight into smaller operators increases kernel boundaries and launch overhead. Third, prefill and autoregressive decode expose different bottlenecks: prefill mainly benefits from reduced arithmetic, whereas decode is memory-bound and repeatedly pays for history-dependent low-rank computation. To address these issues, we introduce FlashSVD v1.5 , a unified high-performance inference runtime for low-rank transformers. It supports multiple public checkpoint families under one execution framework and combines phase-specific execution with dense-KV attention, packed MLP execution, and per-layer CUDA-graph replay to reorganize the fragmented low-rank path into a thinner serving path. Experiments on LLaMA-family decoders show that FlashSVD v1.5 improves decode and end-to-end latency on fixed low-rank checkpoints, with up to 2.55\times decode and 2.39\times end-to-end speedups for autoregressive LLMs. These gains transfer across public checkpoint families and remain visible over long decode horizons. Under practical bf16 cached decoding, FlashSVD remains close to the baseline cached decode path, although we do not claim token-by-token identity. Additional encoder results are reported in the appendix. Together, these results show that making low-rank models actually fast requires system-level co-design, not compression algorithms alone. Figure 1 : Overview of FlashSVD v1.5. We reorganize the fragmented, shattered low-rank execution path (left) into a streamlined, thin serving path (right) via graph replay, eliminating repeated launch bubbles and host bottlenecks. 2 Related Work SVD for Checkpoint Compression. While early works successfully compressed small encoders using standard SVD [ 6 , 9 , 12 ] , modern LLMs require more advanced treatments. Current open-source solutions typically fall into three directions. First, whitening-based methods like ASVD and SVD-LLM apply activation-aware scaling prior to decomposition to ensure truncation is aligned with model performance on down-streaming tasks [ 31 , 26 , 10 , 11 , 13 , 8 , 16 , 25 , 23 , 15 , 17 , 4 , 21 ] . Second, the activation-space truncation methods such as Dobi-SVD, aggressively fit models into extrememly low-ranked activation spaces by exploring the optimal activation subspace in backward propagation and project weights into that low-rank subspace, along with combining SVD with mixed-precision quantization [ 24 , 20 , 1 ] . However, since our goal is to evaluate pure low-rank execution, we consider these quantization-hybrid methods orthogonal to our scope and only consider the activation truncation design and assume strictly uniform-precision baselines. Third, parameter-sharing techniques [ 22 , 17 ] have recently emerged as a highly effective approach. Instead of maintaining independent decomposition factors for each layer, these methods explore a global, shared set of low-rank bases that is reused across multiple Transformer blocks. By decoupling this shared structural basis from layer-specific projections, they drastically reduce the parameter footprint while maintaining model capacity. Attention-Only Decomposition. Parallel to checkpoint compression, another line of research focuses entirely on runtime activations. Methods such as Palu, xKV, and QSVD [ 3 , 2 , 30 , 18 ] apply SVD exclusively to the KV cache while leaving the MLP layers completely uncompressed. This strategy is highly effective for resource-rich, long-context serving where the dynamic KV cache is the absolute memory bottleneck. Our target scenario, however, is fundamentally different. In edge deployments—which are characterized by short contexts and small batch sizes—the static model weights consume the vast majority of memory. Therefore, FlashSVD v1.5 deliberately targets checkpoint compression rather than activation compression, directly tackling the true bottleneck of edge inference. Figure 2 : Transition from low-rank KV factors to dense-KV attention. By materializing historical KV states into a contiguous dense cache, FlashSVD v1.5 avoids the high recomputation and irregular memory access overhead of previous low-rank inference methods. Figure 3 : Reducing kernel fragmentation in MLP blocks. Instead of a dual-split execution, FlashSVD v1.5 utilizes offline prepacking and online packed projection to merge two duplicated paths into one, significantly reducing launch overhead. 3 Methodology FlashSVD v1.5 targets latency-sensitive small-batch LLM serving, especially autoregressive decoding at batch size B=1 . In this regime, low-rank checkpoints often fail to realize their theoretical efficiency not because they lack arithmetic savings, but because the runtime path becomes fragmented. As illustrated in Figure 1, a naive low-rank implementation breaks a Transformer layer into many short-lived projections, reconstructions, cache-handling steps, and kernel launches, creating repeated host interaction and launch bubbles. FlashSVD v1.5 is designed around a simple principle: rather than exposing this shattered execution path directly, we reorganize it into a thin serving path. Concretely, the runtime combines phase-specific execution with three decode-oriented components: dense-KV attention, packed MLP projection, and per-layer graph replay. 3.1 FlashSVD v1.5 Design Unified runtime across checkpoint families. Before execution, FlashSVD v1.5 maps different low-rank checkpoint families, including SVD-LLM v1, SVD-LLM v2, Dobi-SVD, and Basis Sharing, into a common native factorized representation. This normalization allows all supported checkpoints to enter the same runtime and use the same prefill kernels, decode fast paths, and packed-factor caches. For Basis Sharing, we additionally restore true shared Parameter objects across grouped layers so that shared bases remain physically shared and can be packed and reused once. Attention: reconstruct the present once, read the past contiguously. The main bottleneck of low-rank decode attention is that the runtime keeps paying to reconstruct or reinterpret the history. Figure 2 contrasts the two execution styles.