Local Properties of Neural Networks Through the Lens of Layer-wise Hessians

Maxim Bolshim, ITMO University, Alexander Kugaevskikh, ITMO University · 2025

Read Local Properties of Neural Networks Through the Lens of Layer-wise Hessians by Maxim Bolshim, ITMO University, Alexander Kugaevskikh, ITMO University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17486.txt.

Description

Peering inside a neural network layer by layer can reveal why it generalises or fails, and Maxim Bolshim and Alexander Kugaevskikh use the curvature of each layer to do that peering. For every layer they form its Hessian, the matrix of second derivatives that describes how sharply the loss bends around the current settings. The pattern of values in that matrix, its spectrum, carries clues about the layer local behaviour.

Across a large study of many experiments on dozens of data sets, Bolshim and Kugaevskikh find steady regularities in how these spectra evolve during training. The spectra track overfitting, too few parameters, and how expressive a network can be, and they line up with how well the finished model performs. The authors offer the method as a diagnostic tool for designing and steadying deep networks.

Catalog record

CREATORMaxim Bolshim, ITMO University, Alexander Kugaevskikh, ITMO University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17486

Browse the full library · About the Tower of Babel