Why Smaller Is Slower? Dimensional Misalignment in Compressed LLMs
Jihao Xin, Tian Lyu, Qilong Pan, Kesen Wang, Marco Canini · 2026
Read Why Smaller Is Slower? Dimensional Misalignment in Compressed LLMs by Jihao Xin, Tian Lyu, Qilong Pan, Kesen Wang, Marco Canini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09595.txt.
Introduction
Xin and four colleagues name a trap in post-training compression. Shrunken models often run no faster, because ragged tensor shapes clash with the graphics-processor stack. They trace this dimensional misalignment across framework, library, and hardware levels. A compressed Llama-3-8B with 15 percent fewer parameters runs no faster than baseline, since 95 percent of its dimensions misalign. Their GAC wrapper re-selects hardware-aligned dimensions under the same parameter budget, restoring full alignment and winning back up to 1.5 times speedup with quality preserved.
Catalog record
| CREATOR | Jihao Xin, Tian Lyu, Qilong Pan, Kesen Wang, Marco Canini |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28270 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.09595
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Why Smaller Is Slower? Dimensional Misalignment in Compressed LLMs Jihao Xin, Tian Lyu, Qilong Pan, Kesen Wang, Marco Canini Abstract . Post-training compression reduces LLM parameter counts but often produces irregular tensor dimensions that degrade GPU performance—a phenomenon we call dimensional misalignment . We present a full-stack analysis tracing root causes at three levels: framework, library, and hardware. The key insight is that model inference becomes slower because the resulting dimensions are unfriendly with the GPU execution stack. For example, compressing Llama-3-8B with activation-aware singular value decomposition (ASVD) has 15% fewer parameters yet runs no faster than the uncompressed baseline, because 95% of its dimensions are misaligned. We propose GAC (GPU-Aligned Compression), a new compression paradigm that wraps any dimension-reducing compressor and re-selects hardware-aligned dimensions via multi-choice knapsack optimization under the same parameter budget. We evaluate GAC on Llama-3-8B with ASVD and LLM-Pruner, achieving 100% alignment and recovering up to 1.5 × \times speedup while preserving model quality. 1. Introduction Large Language Models (LLMs) deliver strong capabilities, but their scale hinders deployment. Post-training compression reduces model size and can be categorized into three families: quantization ( 5 ; 7 ) , sparsification ( 4 ; 16 ) , and dimension reduction ( 20 ; 2 ; 1 ) . Quantization and sparsification preserve tensor shapes and are therefore compatible with existing GPU kernels. Pretrained models come with carefully designed dimensions that are GPU-friendly, such as 8-aligned dimensions for FlashAttention ( 3 ) . Dimension reduction, however, alters the tensor shapes, often producing irregular dimensions (e.g., dimension reduced from 128 to 107), which conflict with GPU execution primitives—Tensor Core tiles, cuBLAS kernels, and framework backend selection—causing a counterintuitive outcome: models with fewer parameters run slower than uncompressed ones . We term this phenomenon dimensional misalignment . \mathbf{X} M K \cdot \mathbf{W} K N = \mathbf{Y} M N Figure 1 . GEMM: Y{=}X{\cdot}W with X{\in}\mathbb{R}^{M\times K} , W{\in}\mathbb{R}^{K\times N} . To see why, consider one of the core operators in LLMs, GEMM (General Matrix Multiply) in the expression: Y{=}X{\cdot}W (Figure 1 ), with sequence dimension M , inner dimension K , and output dimension N . Three compression methods each target a different axis: (i) low-rank factorization replaces W with two factors A{\cdot}B , introducing a reduced rank r as the inner dimension of two smaller GEMMs ( 20 ; 2 ; 18 ) ; (ii) structured pruning removes neurons or attention heads, shrinking N ( 8 ) ; (iii) token/KV eviction drops sequence entries, reducing M ( 21 ; 1 ) . All three optimize size vs. accuracy without checking whether the resulting dimensions satisfy GPU alignment constraints (e.g., whether d\bmod 8=0 ). The problem is pervasive: ASVD ( 20 ) produces 95% misaligned dimensions, and even LLM-Pruner ( 8 ) —whose pruning granularity is relatively coarse—still leaves 17% of weights misaligned (Table 5 ). Prior hardware-aware methods ( 15 ; 19 ) treat latency as a black-box, tying solutions on specific architectures such as CNNs without isolating root causes or providing parameter-budget guarantees. We propose GAC (GPU-Aligned Compression), a compression paradigm that imposes hardware alignment constraints on any dimension-reducing compressor. GAC first identifies why certain dimensions are slow via a full-stack analysis (§ 3 ); next, it constrains a multi-choice knapsack optimizer to re-select aligned dimensions under the same parameter budget. GAC is a compressor-agnostic framework that restores speed without changing the upstream compressor’s design. Contributions. • We identify the dimensional misalignment problem in compressed LLMs and demonstrate its prevalence across compressors (§ 2 ). • We conduct a systematic full-stack analysis to trace the root causes and categories them into three layers: framework, library, and hardware(§ 3 ). • We formalize the dimension selection as a constrained optimization and provide a dynamic programming solver to guarantee the alignment (§ 4 ). • Preliminary evaluation on Llama-3-8B ( 9 ) achieves up to 1.5 \times speedup without sacrificing accuracy (§ 5 ). 2. Motivation: Dimensional Misalignment Figure 2 . Llama-3-8B at \rho=20\% . Shape: \circ{=} Q Head, \square{=} K Head, \triangle{=} V Head; color: green =8-aligned, red =misaligned. Existing LLM compression aims to preserve accuracy under a given compression ratio \rho . Formally, given the set of pretrained parameters \mathcal{W} , we seek a compressed set \mathcal{W}^{\prime} that minimizes expected loss subject to the size constraint: (1) \min_{\mathcal{W}^{\prime}}\;\mathbb{E}_{(x,y)\sim\mathcal{D}}[\mathcal{L}(\mathcal{W}^{\prime};x,y)]\quad\text{s.t.}\quad 1-|\mathcal{W}^{\prime}|/|\mathcal{W}|\geq\rho. Here \mathcal{D} is the data distribution, \mathcal{L} is the loss, |\mathcal{W}^{\prime}| and |\mathcal{W}| denote total parameter counts. We denote B=(1-\rho)|\mathcal{W}| as the parameter budget. Different parameters have different compression sensitivities, e.g., early and late layers are often more critical than middle layers—so budget cannot be allocated uniformly. Existing methods proceed in two steps. First, for each parameter W_{i} , compute an importance score s_{i} using a proxy (Table 1 ); a higher s_{i} reflects higher sensitivity, so we should retain more parameters in W_{i} . Second, allocate the budget B by assigning each W_{i} a dimension d_{i} so that more important W_{i} receive larger d_{i} : (2) \{d_{i}\}=\arg\max_{\{d_{i}\}}\sum_{i}s_{i}\cdot|W_{i}|\quad\text{s.t.}\quad|\mathcal{W}^{\prime}|\leq B,\quad d_{i}\geq 0. Here d_{i} is the compressed dimension (e.g., inner dimension or output width, varies by compression method), |W_{i}| is the parameter count given dimension d_{i} , |\mathcal{W}^{\prime}|=\sum_{i}|W_{i}| is the total parameter count of the compressed model, and B is the parameter budget. Because s_{i} and the optimum \{d_{i}\} are continuous, the resulting dimensions are typically irregular (e.g., 107, 108, 109) and often violate GPU alignment requirements (e.g. d_{i}\bmod 8=0 ), triggering backend fallbacks and kernel switches that erase the expected speedup from fewer parameters and FLOPs. We demonstrate the dimensional misalignment problem with a real-world example: PaLU ( 2 ) . We use 8-alignment ( d\bmod 8=0 ) as an example of alignment constraints. When dimensions are not multiples of 8, latency can increase by up to 90% (Figure 5 ); we detail this analysis in § 3 . Figure 3 shows that a large fraction of layers end up misaligned (e.g., 78% in this setup). For generality, we summarize the mainstream importance score proxies into four categories (Table 1 ). We empirically measure unconstrained dimension allocation on Llama-3-8B at \rho{=}20\% with all four proxies; Figure 2 demonstrates that misalignment consistently occurs across methods. Appendix A shows the misalignment persists across compression ratios \rho\in[10\%,50\%] . Figure 3 . Llama-3-8B after PaLU compression at \rho=20\% . Table 1 . Importance scores for budget allocation. Method Score Works Magnitude \|W_{i}\|_{F} SVD-LLM ( 18 ) Activation \|X_{i}\|_{F} ASVD ( 20 ) Gradient \left|\frac{\partial\mathcal{L}}{\partial h_{i}}\cdot h_{i}\right| Taylor Pruning ( 10 ) Fisher \mathrm{tr}(\mathbf{F}_{i}) PaLU ( 2 ) 3. Full-Stack Analysis We analyze where and why dimensional misalignment causes slowdowns on an NVIDIA A100-80GB with PyTorch 2.9.1, CUDA 12.8, FP16. Latency is measured with CUDA events (50 warmup, 200 measurement iterations, 3 trials). Root causes fall into three layers (Figure 4 ): Framework, Library, and Hardware. We detail each layer in the following subsections. Framework FlashAttention Math Library cuBLAS CUTLASS Hardware Tensor Core CUDA Core Dispatch Execute Figure 4 . PyTorch SDPA execution stack. 3.1. Framework Layer Existing DL frameworks such as PyTorch and TensorFlow dispatch an operation to different backends. For example, a matrix multiply A@B in PyTorch may run via cuBLAS or via a Triton kernel depending on the shape and hardware, where the