FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation
Ramakrishnan Sathyavageeswaran · 2026
Read FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation by Ramakrishnan Sathyavageeswaran (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16800.txt.
Introduction
Sathyavageeswaran proposes FIM-LoRA, a way to assign LoRA rank per layer by layer informativeness for the task. Before fine-tuning, eight calibration backward passes estimate gradient variance for each LoRA-B matrix, approximating the empirical Fisher diagonal. Rank budget is then redistributed proportionally. The result is a standard LoRA with a per-layer rank pattern, no new parameters, and no serving changes. On GLUE it matches LoRA at the same budget, and on commonsense reasoning it nearly matches LoRA, with memory cost about 256-fold lower than full Fisher.
Catalog record
| CREATOR | Ramakrishnan Sathyavageeswaran |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25426 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16800
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation Ramakrishnan Sathyavageeswaran Abstract Low-rank adaptation (LoRA) assigns a uniform rank to every adapted weight matrix—a practical convenience that ignores a fundamental reality: different layers contribute unequally to task adaptation. We address this with a lightweight engineering solution: before fine-tuning begins, run eight calibration backward passes, compute the gradient variance of each LoRA-B matrix as a proxy for layer informativeness, and redistribute the rank budget proportionally. The resulting adapter is a standard LoRA with a per-layer rank pattern—no new parameters, no training overhead, no changes to serving infrastructure. We implement this via an efficient approximation of the empirical Fisher Information Matrix (eFIM) diagonal, restricted to LoRA adapter matrices (not the full model), which reduces memory cost by ∼ 256 × {\sim}256\times compared to full-model Fisher estimation (for r = 16 r{=}16 , d in = 4096 d_{\text{in}}{=}4096 ). On the GLUE benchmark with DeBERTa-v3-base, FIM-LoRA matches LoRA ( 88.6 88.6 vs. 88.7 88.7 ) at the same parameter budget, and on commonsense reasoning with LLaMA-3-8B reaches 68.5 68.5 vs. 68.7 68.7 for LoRA with our best configuration. Beyond accuracy, the per-layer rank map produced by our method is interpretable: value projections and early-to-middle layers consistently receive higher rank, matching established intuitions about where transformers encode task-relevant semantics. 1 Introduction Fine-tuning a large pre-trained model is now a routine engineering task. Low-Rank Adaptation (LoRA) ( Hu et al. 2022 ) makes it feasible by replacing full-weight updates with low-rank corrections: freeze W\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}} and learn \Delta W=BA where B\in\mathbb{R}^{d_{\text{out}}\times r} , A\in\mathbb{R}^{r\times d_{\text{in}}} , and r\ll\min(d_{\text{out}},d_{\text{in}}) . The standard recipe fixes the same rank r for every adapted matrix in every layer. This is computationally convenient but empirically unjustified: layers differ substantially in how much they need to change for a given task. Assigning equal rank to all layers wastes capacity on layers that barely adapt, while starving layers that carry the bulk of the task signal. The engineering question this paper addresses is: can we cheaply estimate, before training starts, which layers are most informative for a specific task—and allocate rank accordingly? Our approach. We propose a pre-training calibration procedure that estimates per-layer task informativeness in T=8 gradient passes. The key insight is that the gradient variance of a LoRA adapter matrix—formally, the diagonal of the empirical Fisher Information Matrix (eFIM) evaluated at the adapter weights—measures how sensitively the task loss depends on each parameter. Layers with high gradient variance are task-informative : the loss steers them strongly. Layers with low gradient variance are task-redundant : their parameters barely affect task performance. We aggregate this per-element matrix into a single scalar per layer (sample mean), then solve a budget-constrained integer allocation problem to assign ranks. Critically, we compute this signal only on the LoRA-B adapter matrices , not on the full model. This is both principled (the LoRA-A gradient is zero at initialization; see Section 3 ) and practically efficient—it requires no access to billions of base model parameters and runs in the same memory footprint as standard LoRA fine-tuning. Relation to prior FIM-based work. Fisher information has been applied to LoRA in two prior contexts that are orthogonal to ours. Lodha et al. ( Lodha et al. 2023 ) use a full-model FIM score to select which layers to fine-tune at all—a binary, layer-level decision applied to encoder-only models. Kim et al. ( Kim et al. 2025 ) use gradient variance of LoRA adapters to identify parameters encoding data to be forgotten , for the purpose of machine unlearning. Ogawa et al. ( Ogawa et al. 2026 ) use CKA similarity between layer inputs and outputs to select a 50% layer subset to fine-tune. None of these address rank allocation : the continuous, module-level question of how much budget to assign each layer given that all layers participate in fine-tuning. This is our contribution. The FIM diagonal is the signal we use—not the novelty itself; the novelty is the rank allocation algorithm and its engineering properties. Contributions. 1. A calibration-time rank allocation algorithm (FIM-LoRA) that runs in \sim 8 backward passes on LoRA adapter matrices only, producing a standard LoRA with a per-layer rank pattern (Section 3 ). 2. Empirical evidence that task-informative rank allocation matches uniform LoRA on GLUE (8 tasks, 5 methods, 3 seeds, 4 ranks) and is competitive on commonsense reasoning with LLaMA-3-8B (Section 4 ). 3. An ablation demonstrating that the gradient-variance signal is informative: a random-rank baseline with identical budget systematically underperforms, and the minimum-rank floor r_{\text{min}} is the dominant hyperparameter—not the number of calibration passes (Section 5 ). 4. Interpretable per-layer rank maps showing that task-informativeness concentrates in value projections and early-to-middle layers, consistent with transformer interpretability findings (Figure 1 ). We are transparent about where this approach falls short. With r_{\text{min}}=1 , the allocator greedily follows the gradient-variance signal and produces a bimodal rank distribution that costs \sim 1.7 accuracy points on commonsense reasoning relative to uniform LoRA. Setting r_{\text{min}}=8 (half the base rank) recovers the gap, which we interpret as: the signal is informative but must be regularized toward uniformity to avoid over-concentrating rank in a small number of modules. 2 Background and Related Work LoRA. Given a frozen pre-trained matrix W\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}} , LoRA learns \Delta W=BA with A\in\mathbb{R}^{r\times d_{\text{in}}} (Kaiming initialized) and B\in\mathbb{R}^{d_{\text{out}}\times r} (zero initialized). The forward pass becomes h=(W+\frac{\alpha}{r}BA)x . Since B=0 at initialization, the model starts identical to the pre-trained checkpoint. Adaptive-rank methods. AdaLoRA ( Zhang et al. 2023 ) parameterizes adapters as \Delta W=P\Lambda Q and prunes singular values during training. The schedule is finicky (we observed \geq 4\times loss spikes on LLaMA-3-8B) and ships a non-standard adapter format. EVA ( Paischer et al. 2025 ) performs PCA of layer activations on a calibration set and uses the right singular vectors to initialize A and rank the layers—measuring what flows into a layer, not how sensitive the loss is to that layer’s parameters. GoRA ( He et al. 2025 ) uses the gradient nuclear norm of the full weight matrix. ALoRA ( Liu et al. 2024 ) tracks per-rank gradient magnitudes during training. FIM-LoRA differs from all of these: the signal is the gradient variance of the LoRA-B adapter (loss-landscape geometry, not activation geometry), computed pre-training (not during training), restricted to adapter matrices only (not the full model), and the decision is continuous rank allocation (not binary layer selection or per-singular-value pruning). FIM-based methods in adjacent settings. Lodha et al. ( Lodha et al. 2023 ) use a full-model FIM score for layer selection (binary, encoder-only). Their method requires full-model gradient computation and reports an average GLUE drop of 7.68pp vs. LoRA—a substantially higher cost than ours. Kim et al. ( Kim et al. 2025 ) (VILA, COLM 2025) use gradient variance of LoRA adapters to build a forget importance map for machine unlearning—the opposite use case (what to erase vs. what to amplify). Ogawa et al. ( Ogawa et al. 2026 ) use CKA dissimilarity between layer input/output representations to select 50% of layers for fine-tuning, achieving a 0.27pp average GLUE drop with 50% fewer parameters. Our method is orthogonal to all three: we allocate rank budgets across all layers rather than selecting or pruning