Low-rank Momentum Factorization for Memory Efficient Training
Pouria Mahdavinia [email protected], Mehrdad Mahdavi [email protected] · 2025
Read Low-rank Momentum Factorization for Memory Efficient Training by Pouria Mahdavinia [email protected], Mehrdad Mahdavi [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08091.txt.
Description
Pouria Mahdavinia and Mehrdad Mahdavi propose a memory saving training method for large models in a 2025 paper. Fine tuning such models needs far more memory than running them, since optimizer state is large. Past savings routes cut model parts or compress state, but they can lock the training space or need costly full matrix steps.
Their method keeps the training momentum in a split low rank form that updates at each step and closely tracks the full form. Those factors also drive a scaled update step. The authors prove a best rate for the method on standard assumptions. Tests on language model alignment tasks show a strong trade off between memory and quality.
Catalog record
| CREATOR | Pouria Mahdavinia [email protected], Mehrdad Mahdavi [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 108 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08091