Local Properties of Neural Networks Through the Lens of Layer-wise Hessians
Maxim Bolshim, ITMO University, Alexander Kugaevskikh, ITMO University · 2025
Read Local Properties of Neural Networks Through the Lens of Layer-wise Hessians by Maxim Bolshim, ITMO University, Alexander Kugaevskikh, ITMO University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17486.txt.
Description
Peering inside a neural network layer by layer can reveal why it generalises or fails, and Maxim Bolshim and Alexander Kugaevskikh use the curvature of each layer to do that peering. For every layer they form its Hessian, the matrix of second derivatives that describes how sharply the loss bends around the current settings. The pattern of values in that matrix, its spectrum, carries clues about the layer local behaviour.
Across a large study of many experiments on dozens of data sets, Bolshim and Kugaevskikh find steady regularities in how these spectra evolve during training. The spectra track overfitting, too few parameters, and how expressive a network can be, and they line up with how well the finished model performs. The authors offer the method as a diagnostic tool for designing and steadying deep networks.
Catalog record
| CREATOR | Maxim Bolshim, ITMO University, Alexander Kugaevskikh, ITMO University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17486