α \alpha -LoRA: Effective Fine-Tuning via Base Model Rescaling

Aymane El FirdoussiMLO, EPFLEl Mahdi ChaytiMLO, EPFL, Mohamed El Amine SeddikTII, UAEMartin JaggiMLO, EPFL · 2025

Read α \alpha -LoRA: Effective Fine-Tuning via Base Model Rescaling by Aymane El FirdoussiMLO, EPFLEl Mahdi ChaytiMLO, EPFL, Mohamed El Amine SeddikTII, UAEMartin JaggiMLO, EPFL (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21345.txt.

Introduction

Aymane El Firdoussi and three co-authors introduce a new family of fine-tuning methods for adapting pretrained models with little data. The best-known approach, LoRA, adds a small trainable matrix beside a frozen one, and the authors ask whether rescaling the base model itself can make such updates generalize better.

Their rescaling approach is set out as a broader class of reparameterizations that contains the usual additive style as a relative. They prove its value in a high-dimensional binary classification task, using random matrix theory to track how the tuned model behaves. They then check the findings in more realistic settings, including fine-tuning large language models, where the experiments support the theory.

Catalog record

CREATORAymane El FirdoussiMLO, EPFLEl Mahdi ChaytiMLO, EPFL, Mohamed El Amine SeddikTII, UAEMartin JaggiMLO, EPFL
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21345

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

α \alpha -LoRA: Effective Fine-Tuning via Base Model Rescaling Aymane El FirdoussiMLO, EPFLEl Mahdi ChaytiMLO, EPFL, Mohamed El Amine SeddikTII, UAEMartin JaggiMLO, EPFL Abstract Fine-tuning has proven to be highly effective in adapting pre-trained models to perform better on new desired tasks with minimal data samples. Among the most widely used approaches are reparameterization methods, which update a target module by augmenting its frozen weight matrix with an additional trainable weight matrix. The most prominent example is Low Rank Adaption (LoRA) ( Hu et al. 2022 ) , which gained significant attention in recent years. In this paper, we introduce a new class of reparameterization methods for transfer learning, designed to enhance the generalization ability of fine-tuned models. We establish the effectiveness of our approach in a high-dimensional binary classification setting using tools from Random Matrix Theory, and further validate our theoretical findings through more realistic experiments, such as fine-tuning LLMs. 1 Introduction Large foundational models have driven major advances in artificial intelligence across domains such as computer vision and natural language processing. Examples include transformer-based models ( Vaswani et al. 2017 ) operating in natural language domain ( Team et al. 2023 ; Grattafiori et al. 2024 ) or vision domain ( Cordonnier et al. 2020 ; Dosovitskiy et al. 2020 ) . Such models are specifically known for their relatively large size and massive training corpus, which makes them more powerful and adapted for many use cases. However, even with their extensive pre-training, these large models may not excel at some specific tasks without further adjustment. Fine-tuning addresses this need by updating a pre-trained model with task-specific data. Unlike training from scratch, it leverages general pre-trained representations while reducing data and compute requirements. The most common class of fine-tuning methods is Supervised Fine-Tuning (SFT), which relies on labeled data in that adaptation process, and one of its most popular lightweight techniques is Low-Rank Adaptation (LoRA) ( Hu et al. 2022 ) , which updates the desired module by adding a low-rank perturbation to the original (frozen) weight matrix. In this paper, we study fine-tuning through the lens of Random Matrix Theory (RMT), where we introduce a theoretical framework to understand and improve transfer learning. Leveraging the theoretical findings, our key practical idea in the context of LoRA is to scale the frozen weights row-wise with a vector \bm{\alpha} before adaptation, thereby adding a new degree of freedom to the fine-tuning process. We show that this modification leads to an optimal scaling factor \bm{\alpha}^{*} , which is typically different from the standard choice ( \bm{\alpha}=1 ). We analyze this framework in a high-dimensional binary classification setting under a Gaussian Mixture Model, proving the existence of such an optimal \bm{\alpha}^{*} while providing its closed-form expression in terms of scalar data-dependent quantities. We then validate our theoretical insights on real tasks, including transfer learning benchmarks and large language model fine-tuning. Summary of contributions. Our main contributions are summarized as follows: 1. In the context of adaptation fine-tuning (e.g., LoRA), we propose the scaling of the base model weight matrices by a non-trivial row-wise vector \bm{\alpha} . 2. We theoretically prove the existence of an optimal parameter \bm{\alpha}^{*}\neq 1 in high-dimensional binary classification and derive its closed form. 3. We design an algorithm to estimate optimal \bm{\alpha} in complex scenarios such as LLM fine-tuning. 2 Related work Transfer learning foundations. Transfer Learning (TL) studies how knowledge acquired in a source task or domain can be reused to improve learning in a related target task. Early surveys ( Pan & Yang 2009 ; Weiss et al. 2016 ) outlined key settings such as domain adaptation and multitask learning. Most theoretical works established generalization bounds linking transfer success to source error and distributional divergence Ben-David et al. 2010 , and showed how shared representations reduce sample complexity ( Maurer et al. 2016 ; Tripuraneni et al. 2020 ) . More recent studies refined these results under classification and regression settings ( Hanneke & Kpotufe 2024 ; Zhang et al. 2021 ; Klivans et al. 2024 ; Kpotufe & Martinet 2021 ; Cai & Wei 2021 ; Reeve et al. 2021 ) . Fine-tuning pre-trained models. With the advent of large-scale pre-training, fine-tuning has become the dominant strategy for transfer learning. The most popular fine-tuning techniques are Supervised Fine-tuning (SFT) and fine-tuning with Reinforcement Learning (RL). RL-based approaches such as RLHF ( Ouyang et al. 2022 ) , DPO ( Rafailov et al. 2023 ) , GRPO ( Ramesh et al. 2024 ; Guo et al. 2025 ) and their variants are especially popular for reasoning and mathematics tasks, where they often outperform SFT ( Shenfeld et al. 2025 ) . In this paper, however, we focus on SFT techniques. SFT extends the training of a pre-trained model using labeled data. Because these models are typically large in size, it is common to modify only a small fraction of their parameters while leaving most unchanged. This strategy, known as Parameter-Efficient Fine-Tuning (PEFT) ( Xu et al. 2023 ) , aims to achieve strong performance with minimal parameter updates. PEFT methods are usually grouped into three categories: additive, selective, and reparameterized ( Ji et al. 2025 ) . Our work centers on the last category. Reparametrized Fine-tuning. Reparameterization-based fine-tuning adapts a model by expressing its parameters in an alternative form, commonly through a low-rank decomposition, to reduce training costs, while the full weight matrices are reconstructed for inference. The most common technique in this class is Low Rank Adaptation (LoRA) ( Hu et al. 2022 ) , which introduces small, trainable matrices operating alongside the pre-trained weights to inject task-specific updates without burdening the inference process. Many extensions were proposed to enhance the efficiency of LoRA by either acting on the initialization of the low rank modules ( Hayou et al. 2024a ) , their learning rates ( Hayou et al. 2024b ) , normalizing the updates ( Liu et al. 2024 ) , setting adaptive ranks ( Kim et al. 2024 ; Lu et al. 2024 ) , finding optimal placements for LoRA modules ( Hayou et al. 2025 ) , and more ( Zhang et al. 2023b ; Dettmers et al. 2023 ; Kopiczko et al. 2023 ; Zhang et al. 2023a ; Tian et al. 2024 ; Jiang et al. 2024 ) . 3 Problem setting and Background To prove the effectiveness of our new family of fine-tuning algorithms, we will theoretically analyze a binary classification setting under a Gaussian Mixture Model (GMM) using tools from Random Matrix Theory (RMT). Through this analysis, we will prove the existence of an optimal scaling parameter \alpha^{\star} and derive its exact theoretical formulation for these settings. 3.1 Theoretical Setting The goal is to fine-tune a linear classifier, initially pretrained on a dataset called source , in order to perform a target task given a relatively small target data corpus. Pre-training phase. We consider that we are given pairs of pre-training (source) data samples \{(\tilde{\bm{x}}_{i},\tilde{y}_{i})\}_{i=1}^{N} that are distributed, for \tilde{\bm{x}}_{i}\in{\mathcal{C}}_{a} with a\in\{1,2\} , as follows: \displaystyle\tilde{\bm{x}}_{i}\in{\mathcal{C}}_{a}\quad\Leftrightarrow\quad\begin{cases}\tilde{\bm{x}}_{i}={\bm{\mu}}_{a}+\tilde{\bm{z}}_{i},\quad\tilde{\bm{z}}_{i}\sim{\mathcal{N}}({\bm{0}},{\mathbf{I}}_{p}),\\ \tilde{y}_{i}=(-1)^{a}.\end{cases} (1) For convenience and without loss of generality, we further assume that {\bm{\mu}}_{a}=(-1)^{a}{\bm{\mu}} for some vector {\bm{\mu}}\in{\mathbb{R}}^{p} . This setting can be recovered by subtracting \frac{{\bm{\mu}}_{1}+{\bm{\mu}}_{2}}{2} from each data point,

Read the complete text