Efficient Modular Learning through Naive LoRA Summation: Leveraging Orthogonality in High-Dimensional Models
Zhanhao Cao, Clement Truong, Andrew Lizarraga · 2025
Read Efficient Modular Learning through Naive LoRA Summation: Leveraging Orthogonality in High-Dimensional Models by Zhanhao Cao, Clement Truong, Andrew Lizarraga (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11985.txt.
Introduction
Zhanhao Cao, Clement Truong, and Andrew Lizarraga test in 2025 a simple way to combine small fine-tuning modules for language models. Each module, called a LoRA adapter, is trained on a separate topic area. The question is whether modules can just be added together.
Using a small GPT-2 model, they train adapters for math, medicine, and finance questions. Some pairs added together beat training on merged data, while others did worse. Similarity between modules tracked the change in performance. Adding needs no extra training and takes seconds, but interference can appear in larger mixes.
Catalog record
| CREATOR | Zhanhao Cao, Clement Truong, Andrew Lizarraga |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-11985
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Modular Learning through Naive LoRA Summation: Leveraging Orthogonality in High-Dimensional Models Zhanhao Cao, Clement Truong, Andrew Lizarraga Abstract Recent advances in large language models (LLMs) are heavily supported by the size of their parameters, often reaching into the trillions. In parallel, parameter-efficient fine-tuning (PEFT) [ 1 , 2 ] has emerged, allowing models to be fine-tuned by updating only a small fraction of parameters. One such method is Low-Rank Adaptation (LoRA) [ 3 ] , which, in the spirit of principal component analysis, updates only a small low-rank subspace. An important feature of LoRA is that it stores parameter deltas as the product of two small matrices rather than the full parameter dimension. This property enables treating LoRA deltas as efficient building blocks that can be appended to introduce domain expertise. Many prior works addressing interference when merging multiple domain-specific models [ 4 , 5 ] have overlooked a property suggested by the superposition principle: independent matrices are likely to be nearly orthogonal [ 6 ] . Leveraging this insight, we hypothesize that it is practical to train LoRA modules independently on disjoint domains, with their weight differences serving as building blocks that can be instantly appended to introduce specific knowledge through naive addition. Using GPT-2 Small (117M parameters) with LoRA rank 4 and α = 64 \alpha=64 , we fine-tuned adapters for three QA domains (math, medicine, finance). In pairwise tests, adding Math+Medicine adapters improved perplexity by − 9.10 % -9.10\% compared to merged-data fine-tuning, while Math+Finance and Finance+Medicine showed changes of + 4.54 % +4.54\% and + 27.56 % +27.56\% , respectively. Furthermore, the RMS score between the LoRA deltas and the percent change in perplexity score shows a positive and linear relationship. This approach relies solely on simple addition, which can be completed in a few seconds, and achieves comparable performance to models trained on merged data. 1 Introduction While LoRA was originally designed to reduce training and storage costs [ 3 ] , its structure implicitly supports modularity. Specifically, the low-dimension LoRA deltas are modules that can be appended to the base model. This perspective has invited many researchers to see if there are ways to merge multiple LoRA models to create multi-domain systems [ 4 , 5 ] . However, most research on this topic is done using traditional, complex merging techniques. While those works have achieved desirable results, it is at the cost overhead in additional training or time-consuming computations. In contrast, this paper aims to have a naive approach to combine models by only adding the LoRA deltas. The hope is to use the linearity of LoRA updates to answer the question of whether addition could suffice in merging models of near orthogonal domains. If the superposition principle holds empirically for LoRA [ 6 , 7 ] , it would allow for a simple and efficient strategy of training domain-specific LoRA modules in isolation, and use them as building blocks via direct summation. This would lead to a result of multi-domain adaptation without complex merging logic, alignment training, or even accessing the original dataset. This inquiry not only tests the practical feasibility of adapter composition but also sheds light on the structure of learned representations within LoRA. This paper would train multiple models and create all the additive combinations of them to evaluate the orthogonality of different domains, as well as independently analyzing the LoRA deltas to explore its potential orthogonality. 2 Related Work LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks [ 4 ] This work investigates modularity in LoRA by introducing methods to combine multiple skill-specific adapters for complex task composition. Rather than operating directly on LoRA deltas, the authors fine-tune a weighted combination of full models derived from individually trained LoRAs. The paper demonstrates that averaging LoRA-induced full model weights, followed by additional fine-tuning, can outperform traditional model merging techniques. However, the approach requires access to downstream data and introduces additional compute costs during adaptation. Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging [ 8 ] Zheng et al. propose a data-free approach to LoRA merging by explicitly addressing interference among modules. They introduce a method that decouples magnitude and direction in LoRA updates, followed by lightweight optimization steps to enforce mutual orthogonality between components. This ensures minimal overlap and conflict between the merged deltas. The approach outperforms naive addition and prior LoRA merging baselines, especially when the source domains are semantically dissimilar or contain conflicting gradients. Other notable methods • O-LoRA [ 9 ] , which enforces orthogonality at the subspace level to support continual learning without forgetting. • Mixture of LoRA Experts (MoLE) [ 10 ] , which dynamically selects from multiple LoRA modules via a routing mechanism for efficient multi-task adaptation. These works collectively demonstrate possibility of composing LoRA deltas and models. However, most rely on either learned merging strategies or auxiliary optimization procedures. 3 Experimental Setup The experiments uses GPT-2 Small (117M parameters) as the base model. It was chosen for its lower computational cost, with the experiment only asks for relative performances. Additionally, a smaller model is more susceptible to interference when merging LoRA modules due to collision of knowledge domain in smaller dimensions, making it a better candidate for highlighting the strengths and weaknesses of the proposed research. For fine-tuning dataset, we use three domain-specific question-answering (Q&A) dataset: mathematics Q&A data from Amini et al. [ 11 ] , medical Q&A from the medQuad, and financial Q&A from Hugging Face dataset. All dataset is consistent in Q&A format for consistency. It’s worth noting that such similarity might introduce correlation that could lead to constructive interference when summing LoRA deltas together. Each of the dataset is split in approximately 80:10:10 for training, validation, and testing respectively. The validation data is used solely for tuning hyperparameter and reducing overfitting. Using the calculated hyperparameter, the LoRA weights introduced by the training data would serve as the building block to support the experiment as described. Lastly, the testing data would be used to compute perplexity score, which the main metric to compute model capability used in this paper. The same methodologies would also be applied to each combination of datasets for more detailed analysis. The LoRA fine-tuning process is done through the PEFT library, fixing LoRA rank and LoRA alpha in the configuration to ensure consistent delta scale and prevent domain-specific bias when summing the models. Referring to suggested values [ 3 ] , we have selected rank and alpha to be 4 and 64 respectively. These choices of hyperparameter is within the range of typical value, while promotes minimal weight updates in each building block to further pressure the idea of using limited dimension to achieve model training. Additionally, all models are trained and evaluated with maximum sequence length of 64, to standardize input processing. Fixing these hyperparameter is necessary to support unbiased comparison between models. Variables like dropout, batch size, and number of epochs will remain as hyperparameter tuned through validation dataset. The hyperparameter are found through Bayesian grid search technique from the Optuna library [ 12 ] . The models’ training process follows the standard approach, using default GPT-2 tokenizer and default PEFT training configuration, which includes AdamW optimizer and learning rate