Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent

Anherutowa Calvo · 2026

Read Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent by Anherutowa Calvo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02596.txt.

Description

Calvo studies the shape of neural network loss landscapes through a spectral lens. A curvature exponent links Hessian eigenvalues to gradient singular values. That exponent varies by layer type in a clear pattern. It sits near 2 for convolutions and near 1 for transformer attention. MLP up-projections fall below 1. Calvo proves a decomposition that explains the spread. The exponent equals 2 plus a log-slope term. That term measures alignment between Kronecker factor eigenbases and gradient directions.

A spectral transfer identity follows from the split. It ties the curvature exponent to gradient rank decay and Hessian decay. Fits made on separate data recover the Hessian exponent to about 2 percent median error. The test spans 93 layers, five architectures, and three datasets. No free parameters enter the fit. A zeta bound shows curvature piles up near one direction per layer. A proof-of-concept optimizer, Spectral Newton, beats AdamW where the exponent is near 2.

Catalog record

CREATORAnherutowa Calvo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02596

Browse the full library · About the Tower of Babel