Improving Adaptive Moment Optimization via Preconditioner Diagonalization
Son Nguyen, Bo Liu, Lizhang Chen, Qiang Liu · 2025
Read Improving Adaptive Moment Optimization via Preconditioner Diagonalization by Son Nguyen, Bo Liu, Lizhang Chen, Qiang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-07488.txt.
Description
Son Nguyen and three co-authors lift a key training method by a new take on its preconditioner. Methods such as Adam tune each step from stats on past slopes. They hold up well when scale shifts or knobs are off. Yet those stats often miss how slopes move as a pair. That miss can slow the fall of loss in some paths in weight space.
The team tracks paired slope stats in a preconditioner with set form. They map it, with a map that can be run in reverse, to a space where it sits near a pure scale form. A scale form there is cheap to use, with no crude direct guess. Tests show such methods reach low loss faster. For large language models such as LLaMA, training runs about twice as fast as with Adam alone. The method also pairs with lean memory kin such as Adafactor.
Catalog record
| CREATOR | Son Nguyen, Bo Liu, Lizhang Chen, Qiang Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-07488