Refresh-Scaling the Memory of Balanced Adam

Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí · 2026

Read Refresh-Scaling the Memory of Balanced Adam by Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10119.txt.

Description

Fernández-Hernández and four colleagues argue that balanced Adam's remaining parameter is a memory scale, not a fixed constant. Recent evidence shows Adam works robustly when its two momentum parameters are tied, leaving a single parameter, but how to set it is poorly understood. In balanced Adam, it defines a memory horizon, the inverse of one minus beta. The paper studies the refresh count: how often Adam renews its inner stats during useful training.

Across eleven vision and language experiments, choosing beta for a refresh count near one thousand picks different beta values per training scale. Yet it beats the best fixed-beta baseline. Against the strongest fixed choice, beta equals 0.944, the refresh rule cuts the worst-case validation-loss gap by 33.4 percent and brings all eleven runs within one percent of their validation oracle. The result offers a budget-aware view of optimiser scaling.

Catalog record

CREATORAlberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10119

Browse the full library · About the Tower of Babel