𝟙 \mathds{1} LoRA: Summation Compression for Very Low-Rank Adaptation
Alessio Quercia, Zhuo Cao, Arya Bangun, Richard D. Paul, Abigail Morrison, Ira Assent, Hanno Scharr · 2025
Read 𝟙 \mathds{1} LoRA: Summation Compression for Very Low-Rank Adaptation by Alessio Quercia, Zhuo Cao, Arya Bangun, Richard D. Paul, Abigail Morrison, Ira Assent, Hanno Scharr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08333.txt.
Introduction
Alessio Quercia and co-authors squeeze fine tuning down to a single vector per layer in a 2025 paper. Cheap fine tuning methods adjust only a small slice of a big model. The team asked how far that slice can shrink per layer. Their method freezes a simple sum as the squeeze step for each layer.
Only one trainable vector per layer then learns the way back out. Rival methods need more numbers per layer for the same job. Broad tests showed better scores at lower memory and compute cost. Spreading the tuning evenly across layers added a further gain.
Catalog record
| CREATOR | Alessio Quercia, Zhuo Cao, Arya Bangun, Richard D. Paul, Abigail Morrison, Ira Assent, Hanno Scharr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08333
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
𝟙 \mathds{1} LoRA: Summation Compression for Very Low-Rank Adaptation Alessio Quercia, Zhuo Cao, Arya Bangun, Richard D. Paul, Abigail Morrison, Ira Assent, Hanno Scharr Abstract Parameter-Efficient Fine-Tuning (PEFT) methods have transformed the approach to fine-tuning large models for downstream tasks by enabling the adjustment of significantly fewer parameters than those in the original model matrices. In this work, we study the ”very low rank regime”, where we fine-tune the lowest amount of parameters per linear layer for each considered PEFT method. We propose 𝟙 \mathds{1} LoRA (Summation Low-Rank Adaptation), a compute, parameter and memory efficient fine-tuning method which uses the feature sum as fixed compression and a single trainable vector as decompression. Differently from state-of-the-art PEFT methods like LoRA, VeRA, and the recent MoRA, 𝟙 \mathds{1} LoRA uses fewer parameters per layer, reducing the memory footprint and the computational cost. We extensively evaluate our method against state-of-the-art PEFT methods on multiple fine-tuning tasks, and show that our method not only outperforms them, but is also more parameter, memory and computationally efficient. Moreover, thanks to its memory efficiency, 𝟙 \mathds{1} LoRA allows to fine-tune more evenly across layers, instead of focusing on specific ones ( e.g . attention layers), improving performance further. 1 Introduction In recent years, the rapid development of large models has transformed multiple fields in artificial intelligence, especially in natural language processing [ 1 , 33 ] and computer vision [ 14 , 39 , 27 ] . However, adapting these models for specific tasks often demands significant computational power and extensive labeled data, making their deployment challenging for many applications. To mitigate these issues, Parameter Efficient Fine-Tuning (PEFT) methods [ 41 , 36 , 11 , 20 , 12 , 15 ] have emerged as a promising approach to modify pre-trained models with minimal adjustments to their parameters, thereby reducing computational and memory demands while preserving performance. [formula] [formula] [formula] (a) LoRA [formula] [formula] [formula] (b) \mathds{1} LoRA Figure 1 : Comparing our method \mathds{1} LoRA to LoRA. Left: LoRA learns the low-rank decomposition \Delta W=BA , where A\in\mathbb{R}^{r\times k} and B\in\mathbb{R}^{d\times r} . Right: \mathds{1} LoRA replaces the matrices A and B with a sum over the input features x as compression and a learnable vector b\in\mathbb{R}^{1\times d} as decompression: \Delta W=b\mathds{1}^{T} , where the feature sum is \mathds{1}^{T}x=\sum_{i=1}^{k}x_{i} , with \mathds{1} being a vector of length k containing only ones. This reduces the trainable parameters from r\times k+d\times r in LoRA to d per layer for \mathds{1} LoRA. Among various PEFT approaches, BitFit [ 41 ] has gained notable attention. It streamlines fine-tuning by updating only the bias parameters in transformer models, offering a lightweight alternative achieving competitive performance with minimal resource consumption. DiffFit [ 36 ] extends this method by adding a scaling factor to linear layers and also updating normalization layers, enabling rapid adaptation of large pre-trained diffusion models to new domains. These methods significantly speed up training and reduce model storage requirements, allowing practitioners to fine-tune models effectively without extensive retraining of all parameters. Yet, because weight matrices are not changed they may not reach the performance achieved by other methods, such as Low-Rank Adaptation (LoRA) [ 11 , 28 ] . LoRA [ 11 ] and its variants [ 15 , 12 , 20 , 6 , 9 , 19 ] introduce low-rank matrices for model weight updates during fine-tuning, enabling efficient training with fewer parameters, see Figure 1 (left). This approach not only enhances the adaptability of large models but also reduces training time and mitigates overfitting. Compared to BitFit, LoRA achieves stronger performance in certain tasks and demonstrates robust generalization capabilities [ 37 , 31 ] . This performance boost, however, requires more trainable parameters, which has been shown to cause training instability and slower convergence [ 5 ] . Additionally, Pu et al. [ 28 ] observe that LoRA is robust to parameter reduction for text classification and generation tasks, allowing for reductions beyond the original proposed ones. This raises the question: can we maintain or improve performance while reducing LoRA’s trainable parameters to levels comparable to BitFit? In this work, we propose \mathds{1} LoRA, a method centered on very low-rank adaptation, as depicted in Figure 1 . We reduce the parameter count to match that of BitFit [ 41 ] , making it significantly lower than that required by other low-rank adaptation methods. \mathds{1} LoRA applies a predefined non-trainable function for input compression, i.e . the sum over all input features x and utilizes a single trainable vector per layer for decompression. As we demonstrate in our extensive, comparative experiments \mathds{1} LoRA provides enhanced performance and memory efficiency in different tasks. Our contributions can be summarized as follows: • We propose \mathds{1} LoRA, a very low-rank adaptation method which fixes the compression mechanism to a simple feature sum and a vector of d parameters for decompression, where d is the number of output features, significantly reducing the number of trainable parameters. • Due to its low memory footprint at training time, \mathds{1} LoRA enables fine-tuning across more layers in larger models ( e.g . , LLaMA2 7 and 13 billion in our experiments), distributing adjustments across multiple layers instead of focusing solely on limited ones ( e.g . the attention layers), thereby achieving enhanced performance. • \mathds{1} LoRA adds no overhead at inference time, as the update \Delta W can be merged with the initial weight matrix W_{0} in the same way as e.g . done in LoRA [ 11 ] . • Our method is simple to implement, making it accessible for practical deployment. • We conduct extensive empirical evaluations demonstrating that \mathds{1} LoRA outperforms state-of-the-art methods in terms of effectiveness, and parameter, memory and computational efficiencies across multiple tasks and datasets. 2 Related Work PEFT methods aim to adapt large pre-trained models for specific downstream tasks by injecting a small number of learnable parameters while freezing the majority of the model’s parameters. This approach addresses the resource inefficiency associated with full fine-tuning methods, where all model parameters are tuned for the new task. Various strategies are employed to integrate learnable parameters in PEFT. For instance, Adapter [ 10 ] methods introduce small, fine-tunable modules across different layers of Transformer models, optimizing only these new modules for downstream adaptation. Prompt-tuning [ 17 ] and Prefix-tuning [ 18 ] involve inserting a set of trainable tokens into the input layer of Transformers, focusing training efforts on these tokens only. More recently, LoRA [ 11 ] reparameterizes certain pre-trained weights using low-rank matrices, tuning only these additional branches to adapt to new tasks. BitFit [ 41 ] proposes to fine-tune only bias terms, minimizing the number of trainable parameters. Although updating only a fraction of the parameters, these PEFT methods achieve comparable or even superior performance to full fine-tuning approaches. While PEFT methods were initially developed within the natural language processing domain, recent works have extended their application to computer vision tasks. For example, [ 42 ] apply LoRA on the Segment Anything model [ 14 ] , and DiffFit [ 36 ] transfers BitFit to fine-tune the diffusion model. Moreover, PEFT methods have recently gained interest for subspace Bayesian inference in large-scale Transformers [ 23 , 38