Low-rank Momentum Factorization for Memory Efficient Training

Pouria Mahdavinia [email protected], Mehrdad Mahdavi [email protected] · 2025

Read Low-rank Momentum Factorization for Memory Efficient Training by Pouria Mahdavinia [email protected], Mehrdad Mahdavi [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08091.txt.

Description

Pouria Mahdavinia and Mehrdad Mahdavi propose a memory saving training method for large models in a 2025 paper. Fine tuning such models needs far more memory than running them, since optimizer state is large. Past savings routes cut model parts or compress state, but they can lock the training space or need costly full matrix steps.

Their method keeps the training momentum in a split low rank form that updates at each step and closely tracks the full form. Those factors also drive a scaled update step. The authors prove a best rate for the method on standard assumptions. Tests on language model alignment tasks show a strong trade off between memory and quality.

Catalog record

CREATORPouria Mahdavinia [email protected], Mehrdad Mahdavi [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08091

Browse the full library · About the Tower of Babel