Strategic Over-Parameterization for Generalizable Low-Rank Adaptation

Jing Gao 1,2,3 , Zhong-Yi Lu 4 , Pan Zhang 1,2,3 , Ze-Feng Gao 4 † \dagger 1 School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China 2 School of Physical Sciences, University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China 3 CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China 4 School of Physics and Key Laboratory of Quantum State Construction and Manipulation (Ministry of Education), Renmin University of China, Beijing 100872, China [email protected] , [email protected] · 2026

Read Strategic Over-Parameterization for Generalizable Low-Rank Adaptation by Jing Gao 1,2,3 , Zhong-Yi Lu 4 , Pan Zhang 1,2,3 , Ze-Feng Gao 4 † \dagger 1 School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China 2 School of Physical Sciences, University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China 3 CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China 4 School of Physics and Key Laboratory of Quantum State Construction and Manipulation (Ministry of Education), Renmin University of China, Beijing 100872, China [email protected] , [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16470.txt.

Introduction

Gao and three others present LoRA-Over, boosting the tuning landscape during training, then folding it at run time. Fine-tuning is hard, and LoRA limits updates to compact parameters but often loses generalization under transfer across tasks. LoRA-Over injects auxiliary weights into low-rank adapters to broaden the hypothesis space, then folds them into standard low-rank form, keeping run-time cost same as vanilla LoRA. Two plans, one static and one dynamic, direct extra capacity. Across understanding, dialogue, arithmetic, and code benchmarks, LoRA-Over outperforms vanilla LoRA.

Catalog record

CREATORJing Gao 1,2,3 , Zhong-Yi Lu 4 , Pan Zhang 1,2,3 , Ze-Feng Gao 4 † \dagger 1 School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China 2 School of Physical Sciences, University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China 3 CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China 4 School of Physics and Key Laboratory of Quantum State Construction and Manipulation (Ministry of Education), Renmin University of China, Beijing 100872, China [email protected] , [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57115
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16470

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Strategic Over-Parameterization for Generalizable Low-Rank Adaptation Jing Gao 1,2,3 , Zhong-Yi Lu 4 , Pan Zhang 1,2,3 , Ze-Feng Gao 4 † \dagger 1 School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China 2 School of Physical Sciences, University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China 3 CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China 4 School of Physics and Key Laboratory of Quantum State Construction and Manipulation (Ministry of Education), Renmin University of China, Beijing 100872, China [email protected] , [email protected] Abstract Adapting large language models (LLMs) to downstream tasks via full fine-tuning is increasingly impractical due to its computational and memory demands. Parameter-efficient fine-tuning (PEFT) approaches such as Low-Rank Adaptation (LoRA) mitigate this by confining updates to a compact set of trainable parameters, but this aggressive reduction often sacrifices generalization, especially under transfer across heterogeneous tasks and domains. We revisit the tension between parameter efficiency and adaptation capacity, and ask whether the two are truly at odds. We answer in the negative by introducing LoRA-Over, a framework grounded in a simple principle: enrich the optimization landscape during training, then collapse the enrichment at inference. LoRA-Over injects auxiliary parameters into the low-rank adapters during training to broaden the effective hypothesis space, and through a decomposition-based reformulation folds them back into a standard low-rank structure with negligible reconstruction error, keeping inference cost identical to vanilla LoRA. Since not all weight matrices benefit equally from added capacity, we further propose two scheduling strategies, one statically predefined and one dynamically determined at runtime, that direct extra capacity where most needed. We evaluate LoRA-Over on language understanding (GLUE, T5-Base), dialogue (MT-Bench), arithmetic reasoning (GSM8K), and code generation (HumanEval), using LLaMA 2-7B and LLaMA 3.1-8B. Across all benchmarks and scales, LoRA-Over consistently outperforms vanilla LoRA, showing that principled over-parameterization designed to vanish at inference is an effective lever for improving PEFT generalization. Code will be released upon acceptance. 1 Introduction The advent of modern large language models (LLMs) has fundamentally revolutionized the field of machine learning, achieving exceptional performance across a wide range of tasks by leveraging massive pre-training on trillion-token corpora ( Devlin et al. 2019 ; Liu et al. 2019 ; Radford et al. 2019 ; Brown et al. 2020 ) . A dominant strategy for adapting pre-trained LLMs to downstream applications is full-parameter fine-tuning (commonly referred to as full fine-tuning) ( Qiu et al. 2020 ; Raffel et al. 2020 ) , wherein all parameters of the model are updated using gradient-based optimization. While this approach delivers outstanding results, it presents significant challenges, particularly in terms of substantial storage requirements and high computational costs, making its practical deployment challenging. To overcome these limitations, parameter-efficient fine-tuning (PEFT) methods have emerged as a promising solution. PEFT techniques, such as Low-Rank Adaptation (LoRA) ( Hu et al. 2022 ) , aim to efficiently adapt large language models to specific domains by freezing the majority of pre-trained parameters and updating only a small subset of them. This approach alleviates the computational complexity and memory overhead associated with full fine-tuning while maintaining a reasonable level of performance. However, the drastic reduction in trainable parameters can compromise the model’s ability to generalize, often resulting in suboptimal fine-tuning outcomes compared to full fine-tuning. Motivated by this performance gap, we propose strategically over-parameterizing low-rank matrices during fine-tuning to enhance generalization. By utilizing matrix decomposition ( Henry and Hofrichter 1992 ; Tucker 1966 ; Oseledets 2011 ; Gao et al. 2020 ) , we temporarily expand the parameter space during training and re-collapse it for inference, ensuring zero architectural overhead. However, this approach faces the dual challenges of performance degradation resulting from the accumulation of decomposition inaccuracies across deep Transformer layers and the computational redundancy caused by indiscriminately over-parameterizing all matrices regardless of their unequal contributions to task performance ( Zhang et al. 2022 ; Voita et al. 2019 ) . Therefore, identifying and selectively over-parameterizing the most critical matrices is essential. To address these dilemmas, we employ the Matrix Product Operator (MPO) ( Pirvu et al. 2010 ) from quantum many-body physics as our primary method for matrix decomposition, which enables near-lossless matrix reconstruction ( Gao et al. 2020 ) , making it ideal for high-fidelity over-parameterization. Building on this, we introduce LoRA-Over (Over-Parameterization for Low-Rank Adaptation), a framework that adaptively identifies critical matrices through two distinct mechanisms: a predefined strategy that estimates importance via loss sensitivity ( Voita et al. 2019 ) , and a runtime strategy that uses gradient variations ( Hou et al. 2020 ) as a dynamic proxy. By selectively expanding key parameters, LoRA-Over enhances fine-tuning performance across diverse downstream tasks while remaining task-agnostic and inference-efficient. We conduct extensive experiments to evaluate the efficacy of LoRA-Over. For natural language understanding tasks, we assess the performance of T5-Base ( Raffel et al. 2020 ) on the General Language Understanding Evaluation (GLUE) subset ( Wang et al. 2018 ) . For dialogue generation tasks, mathematical reasoning tasks, and code generation tasks, we apply our method to Llama 2-7B ( Touvron et al. 2023 ) and Llama 3.1-8B ( Grattafiori et al. 2024 ) , evaluating their performance on the MT-Bench dataset ( Zheng et al. 2023 ) , the GSM8K dataset ( Cobbe et al. 2021 ) , and the HumanEval dataset ( Chen et al. 2021 ) , respectively. Experimental results demonstrate that LoRA-Over significantly outperforms vanilla LoRA, achieving substantial performance gains. For instance, LoRA-Over consistently outperforms vanilla LoRA by 5.95% on the GLUE subset with T5-Base, and by 0.31, 12.13%, 5.20% on MT-Bench, GSM8K, and HumanEval with Llama 2-7B, respectively. Additionally, LoRA-Over also consistently outperforms vanilla LoRA by 0.11, 6.17%, 1.75% on MT-Bench, GSM8K, and HumanEval with Llama 3.1-8B, respectively. These results underscore the versatility and effectiveness of LoRA-Over in addressing the limitations of vanilla LoRA while preserving computational efficiency. 2 Related work Over-parameterization in learning process. Over-parameterization has demonstrated its effectiveness in multiple dimensions of deep learning. Previous studies have highlighted its utility in improving model initialization ( Arpit and Bengio 2019 ) , optimizing training dynamics through improved convergence ( Allen-Zhu et al. 2019b ; Gao et al. 2021 ; Du et al. 2018 ) , and enhancing generalization capabilities ( Allen-Zhu et al. 2019a ) . Sparked by the hypothesis of lottery theory ( Frankle and Carbin 2018 ) , subsequent research has further emphasized its potential to increase training efficiency ( Malach et al. 2020 ; Pensia et al. 2020 ) and improve model performance ( Chen et al. 2020 ; Brix et al. 2020 ; Prasanna et al. 2020 ) . In particular, in-time over-parameterization strategies ( Liu et al. 2021b ) have been employed to bridge the performance gap between sparse and dense network training regimes. Building on these foundations, we propose to leverage over-parameterization as a principled approach to unlock the latent potential of the LoRA method, boosting its fine-tuning performance. Tensor decomposition in neural network. Tensor decomposition has emerged as a cornerstone technique for improving the efficiency of neural network training and inference. Its versatility is evident in applications including model compression ( Gao et al. 2020

Read the complete text