OP-LoRA: The Blessing of Dimensionality
Piotr Teterwak, Kate Saenko, Bryan A. Plummer, Ser-Nam Lim, University of Central Florida, [email protected] · 2024
Read OP-LoRA: The Blessing of Dimensionality by Piotr Teterwak, Kate Saenko, Bryan A. Plummer, Ser-Nam Lim, University of Central Florida, [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10362.txt.
Introduction
Piotr Teterwak and co-authors improve a popular fine-tuning trick in a 2024 paper. Low-rank adapters, or LoRA, let users tune huge models by training only a small set of added weights, but the training landscape is often badly conditioned, which makes optimization touchy. Earlier fixes used custom optimizers tied to full-model gradients, at high cost and with little flexibility.
Their method, OP-LoRA, replaces each adapter's weights with ones predicted by a small extra network that exists only during training. The extra room lets step sizes adapt on their own, easing the learning-rate guesswork, and the predictor is thrown away afterward, so running the model costs nothing extra. Gains held across small and large tasks, with image generation scores improving by up to fifteen points.
Catalog record
| CREATOR | Piotr Teterwak, Kate Saenko, Bryan A. Plummer, Ser-Nam Lim, University of Central Florida, [email protected] |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10362
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OP-LoRA: The Blessing of Dimensionality Piotr Teterwak, Kate Saenko, Bryan A. Plummer, Ser-Nam Lim, University of Central Florida, [email protected] Abstract Low-rank adapters enable fine-tuning of large models with only a small number of parameters, thus reducing storage costs and minimizing the risk of catastrophic forgetting. However, they often pose optimization challenges, with poor convergence. To overcome these challenges, we introduce an over-parameterized approach that accelerates training without increasing inference costs. This method reparameterizes low-rank adaptation by employing a separate MLP and learned embedding for each layer. The learned embedding is input to the MLP, which generates the adapter parameters. Such overparamaterization has been shown to implicitly function as an adaptive learning rate and momentum, accelerating optimization. At inference time, the MLP can be discarded, leaving behind a standard low-rank adapter. To study the effect of MLP overparameterization on a small yet difficult proxy task, we implement it for matrix factorization, and find it achieves faster convergence and lower final loss. Extending this approach to larger-scale tasks, we observe consistent performance gains across domains. We achieve improvements in vision-language tasks and especially notable increases in image generation, with CMMD scores improving by up to 15 points. Figure 1 : Overparameterizing low-rank adapters at training time improves performance. (a) A standard Low-Rank Adapter learns two rank-reduced matrices (A and B) that are added to the frozen base weights. (b) Our proposed OP-LoRA predicts the adapter weights from an MLP and a learned embedding. (c) Visual results of image generation on Stable Diffusion XL, showing qualitative improvements in generated images using our method compared to standard LoRA, in addition to being more faithful to the text prompt. (d) Performance in vision-language tasks, with OP-LoRA showing accuracy gains over LoRA across training epochs. Figure 2 : Comparison of Mean Squared Error (MSE) loss convergence between matrix factorization with and without MLP-reparameterization. Different learning rates are represented by a blue-to-light blue gradient for MLP models and an orange-to-light orange gradient for standard models. The red line represents the SVD solution. Four MLP solutions reach the reconstruction error achieved with SVD, while none of the standard ones do. 1 Introduction Despite their impressive zero-shot and few-shot capabilities Radford et al. 2021 ; OpenAI 2023 ; Touvron et al. 2023 ; Dubey et al. 2024 , fine-tuning large foundation models for specific tasks can provide significant performance gains but is often compute- and memory-intensive, with risks of catastrophic forgetting Ruiz et al. 2024 ; Cho et al. 2021a ; Biderman et al. 2024 . Methods utilizing low-rank adapters Hu et al. 2022 ; Hayou et al. ; Zhang et al. 2023 ; Liu et al. ; Nikdan et al. 2024 ; Meng et al. 2024 mitigate these challenges by introducing compact, low-rank parameter updates, enabling efficient task adaptation without additional inference costs or large memory demands ( Fig. 1 a). Despite their advantages, low-rank adaptation methods pose significant optimization challenges. These optimization difficulties can result in suboptimal convergence and, potentially, poor task performance. This is observed in practice by Biderman et al. 2024 , where the authors show a large sensitivity to learning rates in LoRA. Prior attempts to address this issue include LoRA+ Hayou et al. , which proposes using separate learning rates for the up-projection and down-projection matrices, though this approach requires careful tuning of both rates. PiSSA Meng et al. 2024 initializes the LoRA weights to the principal components of the weight matrix and removes them from the frozen base weights; however, this breaks the intuition that LoRA serves as a low-rank update to a strong base model, making it instead an update to a weaker model and thus more prone to overfitting. Consequently, there is a need for alternative approaches that retain the benefits of low-rank adaptation while easing optimization. We propose a overparameterization of low-rank adapters using a compact multi-layer perceptron (OP-LoRA). Instead of directly learning low-rank matrices, we introduce an MLP that predicts parameter matrices from a learned input vector ( Fig. 1 b). At first glance, this is a surprising solution. After all, any solution reached with reparameterization could, theoretically, be reached without it. Further, conventional wisdom equates more parameters with more optimization difficulty. However, Arora et al. 2018 show the surprising result that over-parametrization acts as a particularly effective implicit gradient acceleration method in cases where all learned matrices can be collapsed together after training by computing their product. With OP-LoRA, we aim to leverage this theoretical insight by over-parameterizing the low-rank adaptation process in a way that accelerates convergence without altering the expressivity of the learned function. A practical side effect of this over-parametrization is that the MLP’s can be discarded after training, making over-parameterized low-rank adapters just as efficient at inference time as their standard counterparts. We start by studying MLP reparameterization in an intentionally difficult small-scale matrix factorization setting, where we observe faster convergence and reduced sensitivity to hyperparameters. Notably, MLP-overparameterized matrix factorization (OP-MF) often converges to the Singular Value Decomposition(SVD) solution with SGD, a result that standard matrix factorization frequently fails to achieve. Interestingly, overparameterization composes very well with accelerated gradient descent methods such as SGD with Momentum and Adam, indicating acceleration even for the most performant optimizers. Overall, we observe much accelerated convergence even on poorly conditioned problems. Inspired by these initial findings, we apply MLP reparameterization to a diverse range of applications, including image generation, vision-language tasks, and pure language processing. We also re-parameterize DoRA Liu et al. , creating OP-DoRA. Our approach delivers consistently high performance. For example, fine-tuning Stable Diffusion for image generation yields a 15-point ClipMMD Jayasumana et al. 2024 Score improvement on two datasets. These results establish MLP reparameterization as an effective method for fine-tuning, offering substantial gains while remaining lightweight at inference. 2 Related Work Convergence Properties of Neural Networks: Understanding the convergence behavior of neural networks has been a subject of significant research interest. Du et al. 2019 find that gradient descent can find global optima in ResNets. Nguyen and Hein 2017 show that under certain conditions, all local minima are actually also globally optimal. Huang et al. 2020 explore the geometry of loss landscapes through visualization, and find that optimizers are biased to flat minima, which generalize well, due to the volume disparity between wide minima and sharp minima. They reason that in larger dimensional parameter space this disparity is larger, and coin the term the “blessing of dimensionality”. Most relevant to OP-LoRA, though, is Arora et al. 2018 ’s work showing that overparameterization can function as an implicit acceleration mechanism in gradient descent. We leverage Arora et al. 2018 ’s findings in our method design. Low-rank Fine-Tuning: Low-rank fine-tuning, specifically with LoRA (Low-Rank Adaptation) Huh et al. 2023 , has emerged as a powerful approach for adapting pre-trained models with minimal additional parameters. A number of follow ups have emerged, improving performance. AdaLoRA Zhang et al. 2023 prunes weights during training. DoRA Liu et al. adds a magnitude scaling vector to the updated matrix. RoSA Nikdan et al. 2024 adds a sparse weight update to the low-rank update, but requires a full-finetuning pass to compute the weight mask. OLoRA Büyükakyüz 2024 and PiSSA Meng et al. 2024 ease optimization by initializing LoRA orthogonally. However, they