Unbiased Gradient Low-Rank Projection
Rui Pan, Yang Luo, Yuxing Liu, Yang You, Tong Zhang · 2025
Read Unbiased Gradient Low-Rank Projection by Rui Pan, Yang Luo, Yuxing Liu, Yang You, Tong Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17802.txt.
Introduction
Rui Pan and four co-authors fix a bias problem in memory-saving training for large models. Gradient low-rank projection stores only compressed optimizer states during training. GaLore is a well-known method of that kind. Many such methods lack convergence guarantees because projection skews the updates.
The team adds a layerwise sampling step to remove that skew. Their method, called GUM, pairs GaLore's mechanism with the Muon optimizer. They prove GUM matches Muon's convergence guarantees on paper. It keeps the memory savings of low-rank training at the same time. Experiments on fine-tuning and pretraining backed the theory.
Catalog record
| CREATOR | Rui Pan, Yang Luo, Yuxing Liu, Yang You, Tong Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17802
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unbiased Gradient Low-Rank Projection Rui Pan, Yang Luo, Yuxing Liu, Yang You, Tong Zhang Abstract Memory-efficient optimization is critical for training increasingly large language models (LLMs). A popular strategy involves gradient low-rank projection, storing only the projected optimizer states, with GaLore being a representative example. However, a significant drawback of many such methods is their lack of convergence guarantees, as various low-rank projection approaches introduce inherent biases relative to the original optimization algorithms, which contribute to performance gaps compared to full-parameter training. Aiming to tackle this problem, this paper investigates the layerwise sampling technique for debiasing low-rank projection mechanisms. In particular, an instantiation of the paradigm gives rise to a novel and unbiased low-rank optimization method built upon GaLore’s mechanism and the Muon algorithm, named GaLore Unbiased with Muon (GUM). We theoretically prove our method matches the convergence guarantees of the base Muon algorithm while preserving the memory efficiency of low-rank techniques. Empirical experiments on LLM fine-tuning and pretraining also demonstrate non-trivial improvements over GaLore and even better performance than full-parameter training. Further investigation shows that the improvement of this technique comes from a more uniform distribution of knowledge inside layers, leading to more efficient utilization of the model parameter space and better memorization. * * footnotetext: Equal Contribution. 1 Introduction Figure 1: A counterexample of GaLore in linear regression with Muon optimizer ( Jordan et al. 2024 ) , where its debiased version GUM converges while GaLore fails to converge. Large language models (LLMs) have demonstrated impressive performance across a diverse range of tasks, including conversation ( Ouyang et al. 2022 ; Grattafiori et al. 2024a ) , mathematical reasoning ( Guo et al. 2025 ) , and agentic applications ( Qin et al. 2025 ) . The advancement of these powerful LLMs demands substantial GPU memory due to the large size of the underlying models. For example, training a 70B model with full parameters requires approximately 1.2 terabytes of GPU memory, which exceeds the capacity of even 8 \times H100 GPUs. To address this issue, memory-efficient training techniques such as GaLore ( Zhao et al. 2024 ) have been introduced. GaLore projects gradients into a low-rank space, reducing the memory footprint of optimizer states during training. Specifically, it employs the top- r components from Singular Value Decomposition (SVD) to define a compact low-rank space, into which the gradients are projected as R_{t}\leftarrow P_{t}^{\top}G_{t} . The optimization step is then performed in this low-rank space, enabling memory savings for the optimizer states. For example, the first and second moments in Adam ( Kingma and Ba 2014 ) are updated using \tilde{M}_{t}\leftarrow\beta_{1}\tilde{M}_{t-1}+(1-\beta_{1})R_{t} and \tilde{V}_{t}\leftarrow\beta_{2}\tilde{V}_{t-1}+(1-\beta_{2})R_{t} , where the low-rank projected gradient R_{t} replaces the original gradient G_{t} . After the optimization step, the parameter update is projected back to the original space. Algorithm 1 Low-rank projection based gradient descent algorithms 1: Input: Initial weight W_{0}\in\mathbb{R}^{m\times n} (suppose m\leq n ), number of iterations K , learning rate \eta , projection rank r . 2: for t=0 to K-1 do 3: G_{t}=G(W_{t}) \triangleright Obtain the gradient at W_{t} 4: P_{t}\leftarrow\texttt{get\_projector}() \triangleright Obtain the projector P_{t}\in\mathbb{R}^{m\times r} 5: \tilde{G}_{t}=P_{t}^{\top}G_{t} \triangleright Obtain projection of the gradient in the low-rank space 6: S_{t}\leftarrow optimizer.update_state ( \tilde{G}_{t} ) \triangleright Run the base algorithm with \tilde{G}_{t} 7: W_{t+1}=W_{t}-\eta P_{t}S_{t} \triangleright Project the update back and update the weights 8: end for Nevertheless, most low-rank optimization methods introduce biased gradient estimations during training ( Muhamed et al. 2024 ; Zhang et al. 2024a ; He et al. 2024 ; Huang et al. 2025 ) , which can lead to suboptimal convergence behavior and measurable performance gaps compared to standard full-parameter training. These biases arise because low-rank projections, while computationally and memory efficient, do not fully preserve the direction and magnitude of the true gradient, especially in high-dimensional parameter spaces. As a result, the optimization trajectory diverges from that of full-precision training, potentially causing slower convergence, reduced final model quality, or instability in certain regimes ( Zhao et al. 2024 ; Ding et al. 2022 ; Zhang et al. 2024a ; Huang et al. 2025 ) . This limitation is particularly critical when pre-training large language models (LLMs), where even small discrepancies in gradient estimation can propagate and amplify across many layers and iterations. To address this fundamental issue, we investigate the general debiasing technique using layerwise sampling ( Pan et al. 2024 ) , which preserves the memory efficiency of training methods via randomly freezing most of the layers. Specifically, the unique strength of layerwise sampling over the typical low-rank projected algorithms of GaLore is analyzed both theoretically and empirically. The introduction of the debiasing technique into GaLore gives rise to a new algorithm called Galore Unbiased with Muon (GUM) , which demonstrates much better convergence guarantees and practical performance in LLM training tasks. We summarize our major contributions as follows: • We investigate the layerwise-sampling debiasing technique and propose a novel algorithm called G aLore U nbiased with M uon ( GUM ), which unifies the strengths of GaLore and Muon. GUM achieves the same theoretical convergence guarantees as Muon while retaining the memory efficiency of GaLore, enabling scalable and effective training of large models. • Empirical experiments in LLM training demonstrate that GUM consistently outperforms GaLore in instruction-following, mathematical reasoning, and commonsense reasoning tasks under the same memory budget. Surprisingly, in LLM pre-training experiments, GUM even outperforms full-parameter trained AdamW by a non-trivial overall accuracy margin of 0.3%-1.1%, while obtaining on-par or better performance than AdamW in 6 out of 7 tasks. • We analyze the underlying reasoning of GUM’s empirical improvements, discovering that its high-rank update nature leads to a larger overall stable rank and more evenly distributed singular values in model weights, which further induce a more long-tailed activation pattern in trained models. This implies the performance gain is brought by more efficient utilization of the model parameter space, in other words, better memorization. 2 Related Work Parameter-Efficient Algorithms in Practice. Parameter-Efficient Fine-Tuning (PEFT) methods are widely adopted for training large-scale LLMs in practice. A typical approach is LoRA ( Hu et al. 2022 ) , which freezes the original model and attaches a small trainable low-rank adapter, thereby reducing memory consumption and improving training efficiency. However, LoRA has been reported to exhibit a non-trivial performance gap compared to full-parameter training ( Ding et al. 2022 ; Lialin et al. 2023 ) , due to its altered parameter space. These changes in parameter space also introduce theoretical challenges in analyzing LoRA’s convergence properties with respect to the original parameter space. To address the aforementioned deficiencies and extend LoRA to larger-scale training settings, GaLore ( Zhao et al. 2024 ) proposes a different approach, which projects the gradients—rather than the parameters—into low-rank spaces. In doing so, the error between the full gradients and the approximated gradients becomes numerically quantifiable, as they now operate within the same parameter space. Following GaLore, a number of low-rank projection-based algorithms have emerged, where the key component follows a similar paradigm to Algorithm 1 , but with different projection matrices P . GaLore utilizes the top- r entries U[:,:r] from SVD, which is computationally expensive. To address this issue, GRASS ( Muhamed et al. 2024 ) derives a sparse projection