SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates
Rajveer Singh · 2026
Read SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates by Rajveer Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10649.txt.
Introduction
Singh studies the frequency structure of LoRA adaptation, the lightweight fine-tuning method that updates only small weight matrices. A two-dimensional cosine transform of trained updates on BERT and RoBERTa across four language benchmarks shows low frequencies dominate. About a third of coefficients carry 90 percent of the energy. Keeping just 10 percent of coefficients shrinks adapters tenfold, losing under 2 percentage points on SST-2. Masking half the frequencies even beats full LoRA on 3 of 8 model-task pairs.
Catalog record
| CREATOR | Rajveer Singh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18373 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10649
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates Rajveer Singh Abstract We present a systematic empirical study of the spectral structure of LoRA weight updates. Through 2D Discrete Cosine Transform (DCT) analysis of trained adaptation matrices across BERT-base and RoBERTa-base on four GLUE benchmarks (SST-2, MNLI, CoLA, QQP), we establish that LoRA updates are universally dominated by low-frequency components: on average, just 33% of DCT coefficients capture 90% of total spectral energy . Retaining only 10% of frequency coefficients reduces adapter storage by 10 × \times while sacrificing only 1.95pp on SST-2. Notably, frequency masking at k k =50% improves over full LoRA on 3 of 8 model–task pairs, suggesting high-frequency components act as adaptation noise. We further discover that RoBERTa-base is systematically more spectrally compressible than BERT-base across all tasks, and that task complexity governs spectral sensitivity – NLI tasks require more frequency budget than sentiment classification. A subsequent SVD–DCT correlation analysis (Pearson r = 0.906 r=0.906 , p < 10 − 9 p<10^{-9} ) connects the empirical ∼ 33 % \sim 33\% constant to the spectral dynamics of SGD ( Olsen et al. 2025 ) , suggesting a theoretical grounding for this finding. These findings motivate a new design principle for PEFT: spectral sparsity in adaptation . 1. Introduction Low-Rank Adaptation (LoRA) ( Hu et al. 2021 ) has become the dominant paradigm for parameter-efficient fine-tuning (PEFT), reducing trainable parameters to under 1% of full model size by decomposing weight updates into low-rank matrices \Delta W=AB . While LoRA’s effectiveness is well-established, a fundamental question remains unanswered: what is the intrinsic frequency structure of these learned weight updates? In signal processing, natural signals are sparse in the frequency domain – the insight underlying JPEG compression and audio codecs. We hypothesize an analogous principle holds for neural adaptation: task-specific weight updates are spectrally sparse , concentrating in low-frequency DCT components because task adaptation represents smooth, global modifications to pretrained representations. This paper presents the first systematic empirical investigation of this hypothesis. We train LoRA adapters on four GLUE tasks across two model families, extract and analyze their DCT spectra, and characterize the relationship between frequency budget, parameter efficiency, and downstream accuracy. Our study yields four concrete, reproducible findings with direct implications for PEFT design. Research Questions: • RQ1: Do LoRA weight updates concentrate energy in low-frequency DCT components? • RQ2: How much accuracy is preserved when discarding high-frequency coefficients? • RQ3: Does spectral structure vary across tasks and model architectures? • RQ4: Can frequency masking serve as implicit regularization? 2. Background and Related Work 2.1 Low-Rank Adaptation (LoRA) For a pretrained weight W\in\mathbb{R}^{m\times n} , LoRA introduces trainable matrices A\in\mathbb{R}^{m\times r} and B\in\mathbb{R}^{r\times n} with rank r\ll\min(m,n) , learning the update \Delta W=BA . Trainable parameters =r(m+n) , typically 0.1–0.3% of total model parameters. At inference, \Delta W is merged into W at zero additional cost. 2.2 Related PEFT Methods Table 1: Comparison of SpectralLoRA with related PEFT methods. Method Core Idea Gap vs. This Work LoRA ( Hu et al. 2021 ) Low-rank AB decomposition No frequency analysis AdaLoRA ( Zhang et al. 2023 ) SVD-based rank allocation Spatial domain only LoRA-Mini ( Ahmed et al. 2024 ) Decompose + selective training No spectral motivation KronA ( Edalati et al. 2022 ) Kronecker product structure Spatial domain only SpectralLoRA (Ours) DCT analysis of LoRA updates First frequency-domain study 2.3 DCT and Spectral Sparsity The 2D Discrete Cosine Transform (DCT-II) decomposes a matrix into orthogonal frequency basis functions. Low-index coefficients capture global, smooth structure; high-index coefficients capture fine-grained detail. In image compression (JPEG), discarding high-frequency coefficients achieves 10–100 \times compression with minimal perceptual loss because natural images are smooth. We test whether the analogous property holds for neural adaptation matrices. 3. Methodology 3.1 Experimental Setup Table 2: Experimental configuration for all SpectralLoRA analyses. Component Details Models BERT-base-uncased, RoBERTa-base ( \sim 110M parameters each) Tasks SST-2 (sentiment), MNLI (NLI), CoLA (linguistics), QQP (paraphrase) LoRA config r=8 , \alpha=32 , target = [query, value], dropout = 0.1 Train samples 5,000 per task (consistent budget across all experiments) Hardware NVIDIA T4-15GB, PyTorch 2.10, transformers 4.45.2 DCT library scipy.fft.dctn with norm=’ortho’ (2D DCT-II) Seed 42 (all experiments fixed for reproducibility) 3.2 Analysis Pipeline 1. Train LoRA: Standard LoRA training on each task–model pair. 2. Extract \Delta W : Reconstruct \Delta W=B\cdot A for all query/value layers (24 matrices per model). 3. Apply 2D-DCT: F=\mathrm{DCT2D}(\Delta W,\ \texttt{norm='ortho'}) per layer. 4. Energy Analysis: Compute cumulative energy curve; find k\% where 90% energy is captured. 5. Mask & Evaluate: Zero out bottom (100-k)\% coefficients by magnitude; reconstruct \Delta W ; evaluate on validation set. 6. Cross-model/task: Repeat across both models and all 4 tasks. 3.3 Key Formulation F=\mathrm{DCT2D}(\Delta W),\qquad\Delta W_{k}=\mathrm{IDCT2D}(F\odot M_{k}),\qquad M_{k}[i,j]=\mathbf{1}\!\left[\,(i,j)\in\text{top-}k\%\ \text{by}\ |F|\,\right] (1) where M_{k} is a binary mask retaining the top- k\% coefficients by magnitude. Post-hoc compression ratio =(1-k/100)\times LoRA params. At k =10%, this yields a 10\times reduction in adapter storage with the same training cost. 4. Main Results 4.1 Parameter Efficiency BERT-base with LoRA r=8 on query and value projections yields 296,450 trainable parameters (0.27% of total). Table 3 shows SpectralLoRA post-hoc compression at different k\% budgets. Figure 1 shows the accuracy vs. frequency budget curve on SST-2. Table 3: Parameter efficiency of SpectralLoRA on SST-2 (BERT-base). Green = beats LoRA baseline. Trained parameters are identical to LoRA – only storage/deployment footprint is reduced. Method Trained Params Stored Params Reduction SST-2 Full Fine-tuning 109,780,228 109,780,228 1.0 \times \sim 93.0% LoRA r=8 (baseline) 296,450 296,450 1.0 \times 87.73% SpectralLoRA k =50% 296,450 148,225 2.0 \times 88.19% (+0.46) SpectralLoRA k =20% 296,450 59,290 5.0 \times 87.04% ( - 0.69) SpectralLoRA k =10% 296,450 29,645 10.0 \times 85.78% ( - 1.95) SpectralLoRA k =5% 296,450 14,823 20.0 \times 79.82% ( - 7.91) Figure 1 : Accuracy vs. DCT frequency budget on SST-2 (BERT-base). Dashed green line = full LoRA baseline (87.7%). At k =50%, SpectralLoRA exceeds full LoRA by 0.46pp, indicating frequency masking acts as implicit regularization. 4.2 Cross-Task and Cross-Model Results Table 4 presents the full results across 2 models \times 4 tasks. Figure 2 shows per-task accuracy vs. frequency budget curves. Figure 3 shows all curves overlaid for direct comparison. Table 4: Full cross-task, cross-model results. Green = frequency masking improves over full LoRA. Red = high spectral sensitivity. All results on validation sets. Model Task Metric Full LoRA k =10% k =20% k =50% \Delta @10% BERT-base SST-2 Acc 87.73 85.78 87.04 88.19 - 1.95pp BERT-base MNLI Acc 63.50 56.30 61.10 63.40 - 7.20pp BERT-base CoLA Mcc 80.35 75.17 78.10 80.10 - 5.18pp BERT-base QQP F1 72.10 75.44 77.60 75.20 +3.34pp RoBERTa-base SST-2 Acc 92.43 91.40 92.20 92.30 - 1.03pp RoBERTa-base MNLI Acc 76.92 74.66 76.10 76.50 - 2.26pp RoBERTa-base CoLA Mcc 79.00 73.35 76.30 79.40 - 5.66pp RoBERTa-base QQP F1 82.22 79.86 81.40 81.80 - 2.36pp Figure 2 : Per-task accuracy vs. frequency budget for BERT-base (solid circles) and RoBERTa-base (dashed squares) across all four GLUE tasks. Annotations show accuracy at k =10% for each model. RoBERTa consistently recovers accuracy at lower k values, confirming its higher spectral compressibility. Figure 3 : All model–task accuracy vs. frequency budget curves overlaid. BERT-base (solid) and RoBERTa-base