Not How Many, But Which: Parameter Placement in Low-Rank Adaptation
Arijit Sehanobish, Charles Lovering · 2026
Read Not How Many, But Which: Parameter Placement in Low-Rank Adaptation by Arijit Sehanobish, Charles Lovering (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12207.txt.
Introduction
Arijit Sehanobish and Charles Lovering ask whether trained parameters matter in low-rank adaptation. With supervised tuning, random and informed subsets perform comparably. With GRPO, random picks fail to beat the untrained model, while gradient-guided picks recover accuracy. Gradient structure explains the split: supervised gradients are low-rank and stable, so any subset gathers updates, while GRPO gradients spread wide and point near-orthogonally, so only elements keep the signal. Their scoring method finds the parameters in under 10 seconds at under 0.5 percent of cost, on residual-stream projections.
Catalog record
| CREATOR | Arijit Sehanobish, Charles Lovering |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110309 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12207
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Not How Many, But Which: Parameter Placement in Low-Rank Adaptation Arijit Sehanobish, Charles Lovering Abstract We study the parameter placement problem : given a fixed budget of k k trainable entries within the B matrix of a LoRA adapter (A frozen), does the choice of which k k matter? Under supervised fine-tuning, random and informed subsets achieve comparable performance. Under GRPO on base models, random placement fails to improve over the base model, while gradient-informed placement recovers standard LoRA accuracy. This regime dependence traces to gradient structure: SFT gradients are low-rank and directionally stable, so any subset accumulates coherent updates; GRPO gradients are high-rank and near-orthogonal across steps, so only elements with consistently signed gradients retain the learning signal. Our scoring procedure identifies these critical parameters in under 10 seconds at less than 0.5% of training cost. Selected parameters concentrate on residual-stream–writing projections (V, O, Down), stable across model families and scales (1.5B–8B). 1 Introduction Large language models are increasingly ubiquitous, underpinning applications from code generation to scientific reasoning, but adapting them to new tasks remains costly: full fine-tuning updates billions of parameters and requires substantial compute. Parameter-efficient fine-tuning (PEFT) methods address this by training a smaller set of parameters while freezing the rest. Among these, Low-Rank Adaptation (LoRA; 37 ) has become widely adopted, decomposing weight updates into low-rank matrices BA and optimizing only the adapter parameters. Recent works shows that number of trainable parameters in such adapters can be dramatically reduced, via shared random bases ( 44 ) , spectral parameterizations ( 23 ) , learned scaling vectors ( 41 ) , or low-dimensional projections ( 60 ) . These results suggest that strong performance can be achieved with surprisingly small parameter budgets. However, they implicitly assume that performance depends primarily on how many parameters are trained, rather than which ones. We revisit this assumption: Our Research Question We study what we term the Parameter Placement Problem : given a fixed budget of trainable parameters within a LoRA adapter, how does parameter choice affect downstream performance? Concretely, in a standard LoRA layer \mathbf{W}_{0}\mathbf{x}+\mathbf{BAx} , where \mathbf{B}\in\mathbb{R}^{h\times r} is initialized to zero and \mathbf{A}\in\mathbb{R}^{r\times d} is drawn randomly, we keep \mathbf{A} frozen, and restrict training to a subset of k entries of \mathbf{B} . This isolates parameter selection from parameter count, allowing us to ask whether different subsets of equal size behave differently during optimization. Donoway et. al ( 18 ) showed that by searching across hyperparameter sweeps and seeds, subsets of LoRA neurons can yield most of the performance gain. We aimed to directly recover these “circuits”. We made two empirical observations, which align with existing research on the nature of the optimization signal in LoRA ( 70 ) , which together motivate our parameter selection proposal, and explain our results. First, SFT gradients are highly aligned across training steps and admit a low-dimensional structure, so that most parameter subsets receive a coherent update direction. Second, in contrast, RL gradients exhibit substantially higher variance, lower magnitude, and weak alignment across steps. In this regime, many parameter subsets experience updates that largely cancel over time, while a small subset of parameters with consistently large gradients accumulates meaningful updates. Motivated by these observations, we propose a simple gradient-based selection procedure that identifies parameters with consistently high gradient magnitude during a short scoring phase, producing stable subsets across data samples with negligible overhead. To isolate the role of gradient noise, we study a controlled synthetic setting where the signal-to-noise ratio can be varied directly. Consistent with our hypothesis, when gradients are clean, most parameter subsets learn effectively; under high noise, performance depends on selecting parameters aligned with the dominant signal. Our contributions are as follows: • We formalize the parameter placement problem for LoRA, separating the effect of parameter count from parameter identity and showing the critical impact under RL ( Section 3 ). • We propose a lightweight gradient-based selection procedure that identifies effective parameter subsets in seconds and yields competitive performance at extreme parameter budgets ( Section 4 ). • We conduct a detailed investigation on the location, causal effects and the training dynamics of these parameters ( Section 5 ). Organization. Section 2 reviews related work; Section 3 introduces our methodology, and analyzes when placement matters ( Sections 3.3 and 3.5 ); Section 4 presents experiments on fine-tuning, reinforcement learning, and out-of-distribution generalization, followed by a discussion of circuit structure ( Section 5 ). 2 Related Work Our paper touches on several lines of research, discussed below. More background is in Appendix A . Low-rank adaptation and its variants. LoRA ( 37 ) decomposes weight updates into low-rank matrices BA , training only the adapter while freezing the pre-trained model. Subsequent work seeks to improve LoRA along several axes: initialization ( 58 ; 85 ) , training dynamics ( 32 ; 53 ; 41 ) , and rank allocation ( 91 ; 79 ; 17 ) . Instead of modifying LoRA, we study the complementary question: given a fixed adapter, which elements should receive gradient updates. Parameter budget and fixed-basis methods. Several methods reduce adapter size by fixing most parameters and training a small residual. VeRA ( 44 ) shares frozen random A and B across all layers and trains per-layer scaling vectors. FourierFT ( 23 ) represents updates via learnable spectral coefficient over a fixed Fourier basis. RandLoRA ( 3 ) extends VeRA with element-wise learned scaling to recover full-rank expressiveness. LoRA-XS ( 4 ) inserts a tiny r\times r trainable matrix between frozen SVD-initialized factors. Most recently, 60 show that RL can succeed with as few as 13 trainable parameters on instruction-tuned models, using random low-dimensional projections. These methods share a common structure: fix a basis (random, Fourier, SVD) and learn a small set of coefficients. Our approach differs in two respects. First, we do not change the LoRA parameterization, we use standard \mathbf{BA} with a standard optimizer and simply mask which elements of \mathbf{B} receive gradients. Second, our selection is informed: we use gradient statistics from the training distribution to choose elements, rather than relying on a fixed or random basis. Importance scoring and parameter selection. Using gradient or curvature information to identify important parameters has a long history, from Optimal Brain Damage ( 47 ) and Optimal Brain Surgeon ( 31 ) to recent methods like Wanda ( 74 ) and SparseGPT ( 21 ) . Movement pruning ( 68 ) selects parameters during fine-tuning based on gradient-weight products. Other methods (SNIP, GRASP, SynFlow) 48 ; 81 ; 75 addresses issues like layer collapse. The lottery ticket hypothesis ( 20 ) demonstrates that sparse trainable subnetworks exist within dense networks. SPT ( 33 ) uses sensitivity scores to decide where to place adapters; we assume adapters are placed and select which elements within them to train. Our circuit scores are instances of this family, specialized to the LoRA setting where \mathbf{B}=\mathbf{0} at initialization provides a clean starting point for gradient-based scoring. Elicitation and minimal adaptation. 2 show that fine-tuning has low intrinsic dimensionality: randomly projected subspaces of a few hundred dimensions suffice. 18 find that training as few as 10–100 randomly chosen parameters can recover 50% of the performance gap, framing adaptation as elicitation