Towards Symmetric Low-Rank Adapters
Tales Panoutsos, Rodrygo L. T. Santos, Flavio Figueiredo · 2025
Read Towards Symmetric Low-Rank Adapters by Tales Panoutsos, Rodrygo L. T. Santos, Flavio Figueiredo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03719.txt.
Introduction
Tales Panoutsos, Rodrygo Santos, and Flavio Figueiredo introduce, in a 2025 paper, a leaner variant of the low rank adapters widely used to fine tune large models. Standard adapters add a pair of thin matrices whose product updates the model's weights. The pair still holds many numbers, and the new method roughly halves them.
Their adapters, called SymLoRA, force the update matrix to be symmetric. A symmetric matrix can be written with one rotation like factor and a short list of scale values, instead of two separate factors. That form needs about half the fine tuning weights of the standard one. In tests, the authors report negligible loss on downstream tasks compared with the usual adapters. Cheaper fine tuning matters for groups with small compute budgets, and for serving many adapted models at once.
Catalog record
| CREATOR | Tales Panoutsos, Rodrygo L. T. Santos, Flavio Figueiredo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-03719
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Symmetric Low-Rank Adapters Tales Panoutsos, Rodrygo L. T. Santos, Flavio Figueiredo Abstract In this paper, we introduce Symmetric Low-Rank Adapters, an optimized variant of LoRA with even fewer weights. This method utilizes Low-Rank Symmetric Weight Matrices to learn downstream tasks more efficiently. Traditional LoRA accumulates fine-tuning weights with the original pre-trained weights via a Singular Value Decomposition (SVD) like approach, i.e., model weights are fine-tuned via updates of the form B A BA (where B ∈ ℝ n × r B\in\mathds{R}^{n\times r} , A ∈ ℝ r × n A\in\mathds{R}^{r\times n} , and r r is the rank of the merged weight matrix). In contrast, our approach, named SymLoRA, represents fine-tuning weights as a Spectral Decomposition, i.e., Q d i a g ( Λ ) Q T Q\,diag(\Lambda)\,Q^{T} , where Q ∈ ℝ n × r Q\in\mathds{R}^{n\times r} and Λ ∈ ℝ r \Lambda\in\mathds{R}^{r} . SymLoRA requires approximately half of the finetuning weights. Here, we show that this approach has negligible losses in downstream efficacy. Introduction Deploying large Natural Language Processing (NLP) services presents significant challenges. End-users of such applications expect fast response times and seamless, conversation-like interactions. Still, the computational demands of evaluating even a pre-trained model can slow down the system, potentially diminishing the user experience. When considering the environmental impact, running these models continuously for such applications can be costly and environmentally taxing. Finally, large models naturally have higher deployment costs. The cause for such a need are the ever-increasing model sizes in Deep Learning (DL). In fact, there is a belief that larger models may be more suitable for more complex tasks, with NLP tasks being no exception. However, as has been shown by the Pruning Literature ( Blalock et al. 2020 ; Hoefler et al. 2021 ; Wang et al. 2022 ; Wimmer, Mehnert, and Condurache 2023 ) , Deep Neural Networks (NNs) are over-parametrized mainly, with only a fraction of the model weights being needed for accurate predictions. Indeed, this is one of the reasons why efficient pruning is sometimes referred to as finding Lottery Tickets, e.g., finding the model’s weight that matter and only those weights ( Blalock et al. 2020 ) . Correlated to the pruning literature, Adapters ( Houlsby et al. 2019 ; Hu et al. 2022 ; Rebuffi, Bilen, and Vedaldi 2017 ; Yin et al. 2023 ) are a successful training technique used to fine-tune Large Language Models (LLMs) focusing only on a small fraction of weights. Low-rank adapters (or LoRA) have gained significant traction for language tasks. In a few words, LoRA focuses on saving Low-Rank Singular Value Decomposed (SVDs) updates for model weights. The smaller the SVD rank, the fewer the weights stored. Traditional fine-tuning paradigms often involve training part of the parameters or external modules. However, since these methods increase inference latency or reduce the model’s usable sequence length, LoRA became a valuable solution for these two problems while presenting several practical vantages. One of these is that different fine-tuned models share the pre-trained weights, making it possible to build small adapter modules, one for each task, containing only the fine-tuning weights and switch easily between them. In this research paper, we aim at understading if the structure of LoRA updates may be captured by a low-rank Symmetric matrix. In particular, our study focuses on the following hypothesis: May SVD be changed to a Eigen-Decomposition in LoRA? A natural consequence of this choice, is even more reduction in adapter sizers. To investigate our hypothesis, we present SymLoRA, a symmetric version of LoRA focused on Eigendecompositions that reduces the number of fine-tuning weights by half. Reduced model sizes can significantly save storage and operational costs in scenarios where models must be deployed at scale (e.g., large-scale NLP services). The proposed method retains all the advantages of LoRA, as it is designed based on the LoRA paradigm. The main contribution of this work is the introduction of symmetric layers. NNs with symmetric layers have been proven to be universal approximators. They are particularly appealing due to their reduced parameter count, which is half that of a conventional matrix. By applying the appropriate decomposition, we can reparametrize the layer to be symmetric and low-rank, achieving even smaller adapters. We evaluate SymLoRA on the Glue benchmark, finding that it is statistically tied to LoRA. Thus, our approach results in little to no performance loss while effectively reducing the number of trainable parameters. In the next section, we discuss Related Work, followed by our development of SymLoRA, and our Empirical Results. The last section concludes our paper. Related Work This section presents our discussion of related work. It is divided into Low-Rank Adapters, Neural Network Pruning, and Symmetric Neural Networks. Low-Rank Adapters (LoRA) Low-Rank Adapters (LoRA) have gained significant attention due to their efficient fine-tuning methods ( Houlsby et al. 2019 ) , which require only a small number of additional parameters. This approach allows the adapter to be merged directly into the original model layer, enabling adaptation to new downstream tasks without introducing any new layers to the NN. Furthermore, the LoRA method provides the flexibility to split the original layer and the adapter, facilitating seamless transitions between different tasks. The key insight behind LoRA is that the weight updates of a pre-trained model often exhibit an intrinsic low-rank structure. Consequently, only a low-rank decomposition of the update matrix for the original layer needs to be saved and optimized. Section SymLoRA provides the mathematical details of LoRA. Even though there are several extensions of LoRA, to the best of our knowledge, none have looked into Symmetric NNs as we do. Neural Network Pruning Neural network pruning is an active research field with a vast amount of literature. We can point out four recent surveys on the subject ( Blalock et al. 2020 ; Hoefler et al. 2021 ; Wang et al. 2022 ; Wimmer, Mehnert, and Condurache 2023 ) . We refer the reader to these surveys for in-depth reviews. While SymLoRA is not a running method, it effectively reduces the number of adapter weights that need storage. In this sense, SymLoRA shares the same overarching question of running, which is to find the most miniature model for a given task. Our discussion here will be on the importance of pruning and the main ideas related to SymLoRA. Symmetric Networks The third related idea is that of exploring symmetric NNs ( Hu, Zagoruyko, and Komodakis 2019 ) . Here, the NN is enforced to have symmetric layers. By definition, if a network is symmetric, it only needs to store upper or lower triangular matrices, thus saving up computing memory. Hu et al. ( Hu, Zagoruyko, and Komodakis 2019 ) showed that symmetric NNs are universal approximators and thus present a promising starting point for pruning. Similarly, some authors propose favoring Spectral Machine Learning ( Giambagli et al. 2021 ; Buffoni et al. 2022 ) . SymLoRA In this section, we introduce the proposed method, provide a background overview, and highlight the main differences between LoRA and SymLoRA. Given the well-established foundation of the LoRA method, we begin by elaborating on LoRA and follow on to build SymLoRA. Background The LoRA method is