Adam Converges Without Any Modification On Update Rules
Yushun Zhang, Bingran Li, Congliang Chen, Zhi-Quan Luo, Ruoyu Sun · 2026
Read Adam Converges Without Any Modification On Update Rules by Yushun Zhang, Bingran Li, Congliang Chen, Zhi-Quan Luo, Ruoyu Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02092.txt.
Description
Yushun Zhang, Bingran Li, Congliang Chen, and colleagues defend a famous training method in this 2026 paper. Adam is the default trainer for neural networks, including large language models. A known counterexample showed Adam can fail to settle, raising safety doubts. The authors spot a mismatch: the counterexample picks the problem after the settings.
In real work, the problem comes first and settings are tuned to fit. With settings matched to the problem, Adam always settles, they prove. One regime needs a large second setting with the first below its square root. A small second setting leaves a zone where Adam can blow up. The border between zones shifts with batch size, which guides practical tuning.
Catalog record
| CREATOR | Yushun Zhang, Bingran Li, Congliang Chen, Zhi-Quan Luo, Ruoyu Sun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 228877 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.02092