A geometric framework for momentum-based optimizers for low-rank training
Steffen Schotthöfer Timon Klein and Jonas Kusch · 2025
Read A geometric framework for momentum-based optimizers for low-rank training by Steffen Schotthöfer Timon Klein and Jonas Kusch (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17475.txt.
Introduction
Steffen Schotthofer, Timon Klein, and Jonas Kusch give a shape based frame for training slim forms of large neural nets. Slim low rank forms cut cost and storage, but stock training tools can stall on them. The bent shape of the slim space trips plain momentum methods.
Their new tools borrow from methods that track moving slim forms and respect that bent shape. Tests show faster settling and stronger scores at the same size budget. The work gives slim training a firmer math base. It may help labs train big nets on tight gear.
Catalog record
| CREATOR | Steffen Schotthöfer Timon Klein and Jonas Kusch |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-17475
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A geometric framework for momentum-based optimizers for low-rank training Steffen Schotthöfer Timon Klein and Jonas Kusch Abstract Low-rank pre-training and fine-tuning have recently emerged as promising techniques for reducing the computational and storage costs of large neural networks. Training low-rank parameterizations typically relies on conventional optimizers such as heavy ball momentum methods or Adam. In this work, we identify and analyze potential difficulties that these training methods encounter when used to train low-rank parameterizations of weights. In particular, we show that classical momentum methods can struggle to converge to a local optimum due to the geometry of the underlying optimization landscape. To address this, we introduce novel training strategies derived from dynamical low-rank approximation, which explicitly account for the underlying geometric structure. Our approach leverages and combines tools from dynamical low-rank approximation and momentum-based optimization to design optimizers that respect the intrinsic geometry of the parameter space. We validate our methods through numerical experiments, demonstrating faster convergence, and stronger validation metrics at given parameter budgets. 1 Introduction Deep learning models have achieved remarkable success across natural language processing and computer vision tasks, but their deployment remains computationally expensive due to the large number of trainable parameters. To address this, parameter-efficient strategies have been developed to reduce memory and compute requirements during training. Common approaches include sparsification guo2016dynamic ; molchanov2017pruning ; he2017channel , quantization wu2016quantized ; courbariaux2016binarized , and layer factorization. The latter has gained considerable attention for pre-training wang2021pufferfish ; khodak2021initialization ; schotthofer2022low ; schotthöfer2024federateddynamicallowranktraining ; zangrando2023rank and especially fine-tuning hu2021lora ; valipour2023dylora ; zhang2023adalora ; hayou2024lora ; zhao2024galore ; lialin2024relora ; schotthöfer2024GeoLoRAgeometricintegrationparameter . Layer factorization represents weights (or adapters) as low-rank matrices, allowing only the low-rank factors to be trained. This significantly reduces both memory usage and computational cost. In the class of low-rank layer factorizations, one of the most popular methods is LoRA ( hu2021lora ) , which applied typical optimizers such as stochastic gradient descent (sgd) or Adaptive Moment Estimation (Adam) adam directly to the low-rank factors. The combination of these optimizers with LoRA does not guarantee an optimal optimization trajectory schotthöfer2024GeoLoRAgeometricintegrationparameter ; zhang2023adalora . To overcome the former challenge, various improvements have been proposed: For example, LoRA+ hayou2024lora proposes the use of separate learning rates for different components of the low-rank decomposition, Dora ( Mao2024DoRAEP ) normalizes the factor matrices and introduces a magnitude parameter. Furthermore, to overcome the challenge of tuning the rank of the LoRA ansatz, AdaLoRA zhang2023adalora adaptively allocating the parameter budget during training, by masking off rows and columns of the adapter matrices. Other low-rank methods focus entirely on the optimizer states: GaLore zhao2024galore projects full-rank weight gradients into a low-rank subspace to reduce the memory footprint of the optimizer state. Tensor-GaLore tensorgalore generalizes this technique to high-order tensor-parameterized models, further improving efficiency for large-scale architectures. Adafactor adafactor approximates the second-moment matrix using a low-rank decomposition of Adam with low-rank factors. A particularly promising class of layer factorization strategies is dynamical low-rank training (DLRT) which has been introduced in schotthofer2022low and has since been used in various tasks zangrando2023rank ; schotthöfer2024federateddynamicallowranktraining ; schotthöfer2024GeoLoRAgeometricintegrationparameter ; coquelin2024harnessing ; kusch2025augmentedbackwardcorrectedprojectorsplitting . DLRT projects the gradient flow dynamics onto the tangent space of the manifold of low-rank parameters, thereby achieving convergence guarantees to low-rank optimal weights schotthöfer2024GeoLoRAgeometricintegrationparameter . However, these projections are inherently non-smooth, leading to ill-conditioned optimization landscapes and requiring smaller learning rates for stable training schotthofer2022low . To ensure robust integration of such projections, DLRT constrains movement to flat subspaces within the low-rank manifold, enabling stable convergence to low-rank optima. The method is rank adaptive - using a basis augmentation and subsequent singular value-based truncation criterion to adapt the rank of the low-rank factorization. It further enables extensions to increase adversarial robustness of the compressed neural networks by enforcing orthonormality of the low-rank bases and projecting onto a well-conditioned manifold savostianova2023robustlowranktrainingapproximate or regularizing the condition number of the factor matrix schotthöfer2025dynamicallowrankcompressionneural . Despite its efficiency, current DLRT approaches primarily rely on sgd, and it remains unclear how adaptive optimizers such as Adam adam can be effectively applied. This is a significant limitation, as many state-of-the-art models depend on momentum-based optimizers like Adam and its variants for performance and stability. Therefore, the extension of DLRT to such optimization techniques is crucial. To bridge this gap, we introduce a novel momentum-based optimization framework for low-rank pretraining and fine-tuning. The method integrates adaptive momentum techniques into the framework of dynamical low-rank training (DLRT), preserving the low-rank structure of model weights while enabling stable and efficient updates. This establishes a link between low-rank optimization and adaptive gradient methods, yielding both theoretical insights and practical improvements. Beyond the method derivation, we analyze why LoRA-style adapters—and low-rank parameterizations more broadly—require momentum-based optimizers that are aware of the geometry of the low-rank parameter space, i.e., the underlying manifold structure. Naively applying standard optimizers such as heavy ball to low-rank parameterizations can produce updates that do not correspond to a gradient flow leading to a low-rank optimum. As a result, these methods may fail to converge to valid low-rank solutions. We show how DLRT can be adapted to approximate geometry-respecting gradient flows that consistently drive convergence toward low-rank optima. Together, these contributions lay a foundation for more robust and efficient training of large-scale models under low-rank constraints. Compared to low-rank methods that act on the optimizer states only ( zhao2024galore ; tensorgalore ; adafactor ) we provide a wholistic interpretation of a low-rank optimization algorithm that adaptively compresses the network weights, gradients and optimizer states simultaneously, achieving superior compression performance at high validation metrics. We remark that the method is directly extendable for tensor-valued neural networks using, e.g., low-rank Tucker factorization. The paper is structured as follows: We first discuss limitations of naive momentum methods and show how to adapt the underlying gradient flow to achieve convergence to a low-rank optimum in Section 2 . While the adapted gradient flow facilitates convergence, constructing robust numerical optimizers from it is challenging due to its inherent stiffness. We propose a low-rank heavy ball optimizer in Section 3 which integrates the adapted gradient flow robustly. In Section 4 , we construct a fully low-rank Adam optimizer by leveraging insights gained in Section 3 . In Section 5 we underline the efficiency of the proposed method through numerical experiments. In particular, we demonstrate fast convergence and superior validation accuracy at high compression levels for training from scratch, transfer learning, and low-rank finetuning of different neural network architectures and benchmarks. 2 Momentum-based low-rank training We consider a low-rank neural network of the form \displaystyle\mathcal{N}(x)=\sigma_{L}(U_{L}S_{L}V_{L}^{\top}z_{L-1}(x))\,, where z_{L-1}(x) is defined recursively by \displaystyle z_{0}(x)=x\in\mathbb{R}^{n_{0}}\,,\quad\text{and}\quad z_{l}(x)=\sigma_{l}(U_{l}S_{l}V_{l}^{\top}z_{l-1}(x))\in\mathbb{R}^{n_{l}},\quad\forall l \displaystyle=1,\dots,L\,. (1) Here, the weight matrices are defined as W_{l}:=U_{l}S_{l}V_{l}^{\top}\in\mathbb{R}^{n_{l}\times n_{l-1}} , where U_{l}\in\mathbb{R}^{n_{l}\times r_{l}} and V_{l}\in\mathbb{R}^{n_{l-1}\times r_{l}} are orthonormal low-rank factors, and S_{l}\in\mathbb{R}^{r_{l}\times r_{l}} is the coefficient matrix. Thus, W_{l} lies in the manifold of rank r_{l} matrices which we denote by \mathcal{M}_{r_{l}} . Additionally, \sigma_{l} represents the activation function of layer l . For simplicity of notation, we do not consider biases, but a model with biases can always be expressed as ( 1 ) by folding biases into weights and creating an input dimension that is always one. Several methods have been proposed to train the low-rank weight matrices W_{l}