Refresh-Scaling the Memory of Balanced Adam
Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí · 2026
Read Refresh-Scaling the Memory of Balanced Adam by Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10119.txt.
Description
Fernández-Hernández and four colleagues argue that balanced Adam's remaining parameter is a memory scale, not a fixed constant. Recent evidence shows Adam works robustly when its two momentum parameters are tied, leaving a single parameter, but how to set it is poorly understood. In balanced Adam, it defines a memory horizon, the inverse of one minus beta. The paper studies the refresh count: how often Adam renews its inner stats during useful training.
Across eleven vision and language experiments, choosing beta for a refresh count near one thousand picks different beta values per training scale. Yet it beats the best fixed-beta baseline. Against the strongest fixed choice, beta equals 0.944, the refresh rule cuts the worst-case validation-loss gap by 33.4 percent and brings all eleven runs within one percent of their validation oracle. The result offers a budget-aware view of optimiser scaling.
Catalog record
| CREATOR | Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10119