Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization

Akiyoshi Tomihari [email protected], Issei Sato [email protected] · 2025

Read Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization by Akiyoshi Tomihari [email protected], Issei Sato [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00213.txt.

Description

Akiyoshi Tomihari and Issei Sato explain why adaptive optimizers beat plain gradient descent on transformers in a 2025 paper. Transformers train poorly with standard stochastic gradient descent and lean on Adam instead. The reason was unclear. The authors study gradient heterogeneity: how wildly gradient sizes vary across blocks of parameters.

Theory shows this unevenness, together with curvature unevenness, slows plain descent. Sign-based updates, which move each parameter by a fixed step in the gradient's direction, shrug it off. Adam behaves like those sign methods, which explains its edge. The team traces the unevenness to layer normalization placement, worst in post-normalization designs. Fine-tuning tests in language and vision backed the theory. They also derive how learning rates should scale for sign methods. The paper offers a cleaner story for a daily puzzle in deep learning.

Catalog record

CREATORAkiyoshi Tomihari [email protected], Issei Sato [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE153 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-00213

Browse the full library · About the Tower of Babel