Adam Converges Without Any Modification On Update Rules

Yushun Zhang, Bingran Li, Congliang Chen, Zhi-Quan Luo, Ruoyu Sun · 2026

Read Adam Converges Without Any Modification On Update Rules by Yushun Zhang, Bingran Li, Congliang Chen, Zhi-Quan Luo, Ruoyu Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02092.txt.

Description

Yushun Zhang, Bingran Li, Congliang Chen, and colleagues defend a famous training method in this 2026 paper. Adam is the default trainer for neural networks, including large language models. A known counterexample showed Adam can fail to settle, raising safety doubts. The authors spot a mismatch: the counterexample picks the problem after the settings.

In real work, the problem comes first and settings are tuned to fit. With settings matched to the problem, Adam always settles, they prove. One regime needs a large second setting with the first below its square root. A small second setting leaves a zone where Adam can blow up. The border between zones shifts with batch size, which guides practical tuning.

Catalog record

CREATORYushun Zhang, Bingran Li, Congliang Chen, Zhi-Quan Luo, Ruoyu Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE228877
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.02092

Browse the full library · About the Tower of Babel