Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent
Anherutowa Calvo · 2026
Read Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent by Anherutowa Calvo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02596.txt.
Description
Calvo studies the shape of neural network loss landscapes through a spectral lens. A curvature exponent links Hessian eigenvalues to gradient singular values. That exponent varies by layer type in a clear pattern. It sits near 2 for convolutions and near 1 for transformer attention. MLP up-projections fall below 1. Calvo proves a decomposition that explains the spread. The exponent equals 2 plus a log-slope term. That term measures alignment between Kronecker factor eigenbases and gradient directions.
A spectral transfer identity follows from the split. It ties the curvature exponent to gradient rank decay and Hessian decay. Fits made on separate data recover the Hessian exponent to about 2 percent median error. The test spans 93 layers, five architectures, and three datasets. No free parameters enter the fit. A zeta bound shows curvature piles up near one direction per layer. A proof-of-concept optimizer, Spectral Newton, beats AdamW where the exponent is near 2.
Catalog record
| CREATOR | Anherutowa Calvo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.02596