LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning
Zihang Liu, Tianyu Pang, Oleg Balabanov, Chaoqun Yang, Tianjin Huang, Lu Yin, Yaoqing Yang, Shiwei Liu · 2025
Read LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning by Zihang Liu, Tianyu Pang, Oleg Balabanov, Chaoqun Yang, Tianjin Huang, Lu Yin, Yaoqing Yang, Shiwei Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00772.txt.
Introduction
Zihang Liu and co-authors find a better way to pick weights for fine-tuning language models in a 2025 paper. Tuning on small, high-quality sets can give strong reasoning, but full tuning is costly and can overfit or erase old skills. Sparse tuning updates few weights, yet picking the right ones for reasoning was hard.
The authors show that after a low-rank reduction, the largest remaining weights are the ones that matter. Tuning only those principal weights worked far better than the same pick without reduction. The result is a cheaper, safer route to reasoning gains when data are scarce. It may help small teams adapt big models.
Catalog record
| CREATOR | Zihang Liu, Tianyu Pang, Oleg Balabanov, Chaoqun Yang, Tianjin Huang, Lu Yin, Yaoqing Yang, Shiwei Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00772
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning Zihang Liu, Tianyu Pang, Oleg Balabanov, Chaoqun Yang, Tianjin Huang, Lu Yin, Yaoqing Yang, Shiwei Liu Abstract Recent studies have shown that supervised fine-tuning of LLMs on a small number of high-quality datasets can yield strong reasoning capabilities. However, full fine-tuning (Full FT), while powerful, is computationally expensive and susceptible to overfitting and catastrophic forgetting, particularly when data is limited. Sparse fine-tuning, which previously achieved notable success by updating only a small subset of model parameters, offers a promising trade-off between efficiency and effectiveness. Yet, it has lagged behind in the LLM era due to the difficulty of identifying parameters truly critical for reasoning. In this work, we state that weights with the largest magnitude after low-rank approximation are critical weights for fine-tuning, which we call Principal Weights . Surprisingly, while magnitude-based sparse fine-tuning performs poorly as a baseline on LLM fine-tuning, it becomes highly effective after rank reduction. These insights motivate our method: L ow-rank I nformed Sparse F ine- T uning ( LIFT ). LIFT only updates the top 5% Principal Weights throughout training and consistently achieves better performance on reasoning tasks than Full FT, while maintaining memory efficiency on par with popular parameter-efficient fine-tuning methods. In addition to strong performance on target domains such as arithmetic reasoning, LIFT also retains up to 20% more source-domain knowledge, compared to Full FT and LoRA. Our code is available at: https://github.com/zihanghliu/LIFT . Figure 1 : Overview of LIFT . LIFT first performs SVD on the original weight matrix W to obtain a Rank-r approximation W^{\prime} . It then selects the top-K parameters of W^{\prime} with the highest magnitudes to create a fine-tuning mask. This mask is then applied to the original weight matrix W for fine-tuning. 1 Introduction Large language models have recently undergone a revolutionary advancement in reasoning capabilities through Supervised Fine-Tuning (SFT) ( Ye et al. 2025 ; Muennighoff et al. 2025 ) and Reinforcement Learning (RL) ( Guo et al. 2025 ; Face 2025 ) . Performing SFT on a small, high-quality dataset delivers remarkable reasoning performance on math problems ( Muennighoff et al. 2025 ) . However, Full Fine-Tuning (Full FT) is prone to overfitting on limited training data ( Chu et al. 2025 ) and incurs substantial computational costs due to the massive sizes of modern LLMs ( Yin et al. 2023 ) . On the other hand, Sparse Fine-Tuning (Sparse FT) ( Guo et al. 2020 ; Xu et al. 2021 ; Sung et al. 2021 ; Sanh et al. 2020a ) , a standout approach for pre-LLM fine-tuning, has demonstrated promising performance by training only a small subset of the base model’s parameters. However, the adoption of Sparse FT has significantly lagged behind its low-rank counterparts in LLMs, as it struggles to identify parameters truly critical to fine-tuning, and its memory overhead is the same as Full FT with irregular sparse patterns. In this paper, we propose L ow-rank I nformed Sparse F ine- T uning ( LIFT ), an effective and efficient approach for reasoning-focused LLM fine-tuning. LIFT builds on a counter-intuitive finding: the most naive baseline for Sparse FT, i.e., magnitude-based fine-tuning, becomes remarkably effective after applying low-rank approximation. We hence identify the weights with the largest magnitude after rank reduction as Principal Weights . The process of obtaining Principal Weights is illustrated in Figure 1 . Empirical results show that LIFT outperforms state-of-the-art PEFT methods, Sparse FT methods, and Full FT on a wide range of tasks. LIFT solves the challenges of Sparse FT in these ways: Prior Knowledge: LIFT identifies Principal Weights , which are critical for retaining pre-training knowledge and adapting to downstream tasks. The intuition aligns with recent findings that reasoning capacity is already in base models ( Ye et al. 2025 ; Yue et al. 2025 ) . LIFT further finds that this knowledge is encoded with Principal Weights and fine-tuning only these parameters is sufficient to achieve comparable—or even superior—reasoning performance. Memory Efficiency: LIFT offers significantly better memory efficiency than Full FT and is on par with LoRA. LIFT updates and stores only a small subset of parameters during fine-tuning, leading to substantial memory savings—particularly in optimizer states, which are reduced from 27GB in Full FT to just 1.3GB ( < 5%) on LLaMA-2-7B. Our analysis reveals that Principal Weights are more important for LLM fine-tuning compared to other weight selection criteria: Adding random perturbation to Principal Weights drastically affects model performance, substantially greater than other sparse selection metrics, both for pre-training knowledge and downstream tasks. In addition, the update matrix of LIFT has a substantially larger magnitude than LoRA and Full FT, and a substantially larger rank than that of LoRA, close to Full FT, enabling a larger capacity to acquire new knowledge in fine-tuning. Furthermore, LIFT can strongly affect the principal eigenspace of the LLM, making a significantly larger deviation than LoRA and Full FT, leading to better adaptation of the downstream tasks. We summarize our contributions as follows: (a) Wikitext Perplexity ( \downarrow ) (b) Next-token Prediction ( \uparrow ) (c) Arithmetic Reasoning ( \uparrow ) Figure 2 : Evaluating pre-trained LLaMA-2-7B model with random noise added to selected parameters. • We propose LIFT , a memory-efficient Sparse Fine-tuning algorithm, that selects and fine-tunes Principal Weights , as parameters with the highest magnitude after low-rank approximation. LIFT has significantly lower memory overhead than Full FT (less than 5% memory for optimizer), similar to that of LoRA. We show that Principal Weights are crucial for retaining pre-trained knowledge and adapting to downstream tasks. • We show that LIFT consistently yields strong performance on diverse sets of tasks. We evaluate LIFT on a wide range of reasoning benchmarks, including GPQA Diamond, Commonsense Reasoning, Arithmetic Reasoning, and Natural Language Understanding. We show that LIFT outperforms state-of-the-art PEFT methods, Full FT, and other Sparse FT strategies. Specifically, LIFT achieves up to 4.42% better performance than LoRA on commonsense reasoning, and up to 2.02% higher overall performance than Full FT on GPQA Diamond. • We provide a comprehensive analysis of LIFT , and show that LIFT has 1) Strong generalization performance , that it balances learning and forgetting, achieving stronger performance on target domains while up to 20% better than LoRA on source domains; 2) Strong learning capacity , that it enables larger weight updates, with rank significantly higher than LoRA, close to Full FT, as well as other fascinating observations. 2 Related Works Low-rank Approximation of Weight Matrices. Recent works ( Sharma et al. 2024 ; Chen et al. 2025 ; Wang et al. 2024 ; Wei et al. 2024 ; Jaiswal et al. 2024 ; Geva et al. 2023 ) study the low-rank structure within the weight matrices of LLMs. Specifically, Sharma et al. 2024 and Chen et al. 2025 found that low-rank approximation of the feedforward layer can improve reasoning capabilities. Sharma et al. 2024 proposes that higher-order components of a weight matrix can introduce noise in decision-making, and eliminating the higher-order components “denoises” the model and helps recover “hidden”, less-frequent information, improving the model’s performance on questions whose answers are supported by less frequently occurring data. Chen