Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning

Zhi-Quan Feng, Ying-Jia Lin, Hung-Yu Kao · 2026

Read Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning by Zhi-Quan Feng, Ying-Jia Lin, Hung-Yu Kao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01046.txt.

Introduction

Zhi-Quan Feng, Ying-Jia Lin, and Hung-Yu Kao improve how LoRA adapters for large language models are initialized. LoRA restricts updates to low-rank subspaces, cutting training cost, but success depends heavily on which subspace is chosen at the start. Existing strategies rely mainly on the geometry of pretrained weights, assuming weight structure alone reflects task relevance, and they ignore how the model interacts with downstream data.

The authors formulate initialization as finding directions in parameter space with the greatest impact under the target data distribution. Their Fisher-guided framework uses curvature information from downstream data to select directions. Across diverse tasks and modalities, this data-aware initialization consistently and significantly improves downstream performance over existing approaches.

Catalog record

CREATORZhi-Quan Feng, Ying-Jia Lin, Hung-Yu Kao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01046

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning Zhi-Quan Feng, Ying-Jia Lin, Hung-Yu Kao Abstract LoRA adapts large language models (LLMs) by restricting updates to low-rank subspaces of pre-trained weights. While this substantially reduces training cost, the effectiveness of adaptation critically depends on which subspace is chosen at initialization: a poor initialization that allocates capacity to task-irrelevant directions can severely hinder downstream performance. Existing initialization strategies primarily rely on the intrinsic properties of pre-trained weights, implicitly assuming that weight geometry alone reflects task relevance. However, such criteria overlook how the model interacts with the downstream data distribution. In this work, we formulate LoRA initialization as identifying the degree of impact of directions in parameter space under the target data distribution. We argue that data-aware sensitivity, rather than weight-only magnitude, should govern the choice of adaptation subspaces. Building on this perspective, we propose a Fisher-guided framework that leverages curvature information induced by downstream data to characterize how parameter perturbations influence model predictions. This perspective yields a principled, task-dependent criterion for selecting LoRA directions that better align adaptation with the target objective. Empirical results across diverse tasks and modalities demonstrate that data-aware initialization consistently and significantly improves downstream performance over existing approaches. 1 Introduction Parameter-Efficient Fine-Tuning (PEFT) methods, especially LoRA, offer a practical alternative by injecting small, trainable low-rank adapters into frozen weights of large language models (LLMs) to reduce the computation cost of training LLMs. Beyond reducing parameter counts, the effectiveness of such adapters depends critically on which subspace of the original weight space they occupy: a well-chosen initialization can concentrate learning on task-relevant degrees of freedom and drastically improve downstream task convergence as well as final accuracy. Most advanced LoRA initialization strategies are grounded in the geometric structure of pre-trained weights, often operationalized through singular value decomposition (SVD) of the corresponding weight matrices ( 19 ; 33 ; 32 ) . While spectral decompositions of pre-trained weights reveal important structural priors, they are blind to the downstream data distribution and the sensitivity of the model to that data. Recent empirical studies show that the singular directions most useful for one task can be uninformative for another ( 9 ) . In short, weight-only criteria risk allocating scarce adaptation capacity to directions that are passive under the target data distribution. A more principled approach is to select adaptation directions using data-aware statistics that quantify how parameter perturbations influence the model’s outputs on the target task. We define Fisher energy based on second-order derivatives of the weight matrices, and our observation shows that Fisher Energy provides an effective measure of parameter sensitivity with respect to the data distribution, and that prioritizing directions with lower Fisher Energy leads to improved downstream performance. Motivated by this observation, we introduce FILet ( F isher-Guided I nitialization for L oRA Fin e-T uning), a data-aware framework that leverages Fisher Energy to identify and align LoRA adaptations with convergence-favorable directions in parameter space. Specifically, FILet computes the second order derivation using Kronecker-factored empirical covariances, which enables efficient capture of data-induced parameter sensitivity without incurring significant computational overhead. Fisher Energy is then used to guide the initialization of LoRA factors, constraining parameter updates to Fisher-aligned subspaces. We comprehensively evaluate FILet against existing SVD-based ( 19 ; 33 ; 32 ) and data-driven LoRA initialization methods ( 28 ) across natural language reasoning, image classification, and natural language generation tasks on large language models. Experimental results demonstrate that this data-aware initialization strategy consistently and significantly improves overall performance across diverse downstream tasks. 2 Related Work LoRA ( 13 ) has emerged as one of the most popular and powerful PEFT techniques for LLMs. By introducing trainable low-rank matrices to approximate weight updates, LoRA substantially reduces the number of trainable parameters while maintaining performance comparable to full fine-tuning. Building upon this foundation, numerous LoRA variants have been proposed to further enhance adaptability and efficiency from various perspectives, including dynamic rank adaptation ( 41 ) , rank expansion through nonlinear transformations ( 14 ) , reduction of trainable parameters ( 15 ; 42 ) , multitask learning extensions ( 36 ; 40 ) , and integration with mixture-of-experts (MoE) architectures ( 30 ; 38 ; 9 ) . One of the most actively explored research directions in LoRA-based fine-tuning concerns the adaptation direction of low-rank modules. DoRA ( 17 ) first highlighted this issue by decomposing LoRA updates into magnitude and direction components, demonstrating that tuning only the directional component can yield more stable and efficient adaptation. Other methods such as PiSSA ( 19 ) , MiLoRA ( 33 ) , CorDA ( 37 ) , and KaSA ( 32 ) further investigate direction-aware adaptation by decomposing pre-trained weights via SVD. These approaches define and select the most important singular directions to initialize the LoRA parameters, while retaining other directions in the frozen residual weights. Suppose a weight matrix W_{0} has the SVD decomposition W_{0}=U\Sigma V^{\top} , where U and V are orthonormal matrices and \Sigma is a diagonal matrix of singular values. The LoRA Modules A\in\mathbb{R}^{r\times n} and B\in\mathbb{R}^{m\times r} are then initialized as: \displaystyle A=\sqrt{\Sigma^{\prime}}\,V^{\prime\top},\;B=U^{\prime}\,\sqrt{\Sigma^{\prime}},\;W_{\text{res}}=W_{0}-AB, (1) where U^{\prime} , V^{\prime} , and \Sigma^{\prime} correspond to the selected singular directions and values based on the specific criteria of each method. W_{\text{res}} denotes the residual weights and W_{0} denotes the pre-trained weights. PiSSA focuses on fine-tuning the most principal singular directions, capturing the most informative subspace of the pre-trained weights. In contrast, MiLoRA targets the least dominant singular directions, emphasizing the underrepresented components ignored during pre-training. KaSA further refines this idea by discarding the least dominant directions to reduce noise, and selectively training on moderately weak directions that preserve informative variability for LoRA adaptation. However, a key limitation of these approaches lies in their exclusive focus on interpreting the spectral properties of pre-trained weights, which may not faithfully capture the model’s sensitivity to data. This raises a natural question: Why not directly ask the data? In other words, downstream data-aware method is more principled than these heuristic weight-only criteria for selecting adaptation directions. Current data-aware methods, such as LoRA-Dash ( 28 ) , proposes a training-time enhancement for LoRA: it performs a short warm-up fine-tuning, then applies SVD to the LoRA weights to extract task-specific directions (TSD) for reparameterization, aiming to accelerate convergence and improve performance. LoRA-GA ( 34 ) and EVA ( 23 ) also leverage data-driven statistics to guide LoRA adaptation. LoRA-GA aligns low-rank updates with full fine-tuning gradients via direction selection and scaling while EVA performs SVD on minibatch activation vectors and emphasizes explained variance and rank reallocation. 3 Method In this section, we introduce the Fisher-Guided LoRA Initialization framework. Our goal is to construct low-rank adaptation matrices whose initialization directions are primarily determined by data sensitivity rather than singular values. We present the problem definition in 3.1 Problem Definition and our preliminary experiments for this design in 3.2 Preliminary . Our method, illustrated in Figure 2 , is introduced in Section 3.3 Fisher-Guided Initialization . Appendix A further summarizes the symbols and notations used throughout the paper, presents the algorithm pseudocode, and provides additional mathematical derivations and proofs. 3.1 Problem Definition The LoRA initialization problem can be formulated as partitioning the pretrained weight space

Read the complete text