Generalization Bounds for Rank-sparse Neural Networks
Antoine Ledent, Rodrigo Alves, Yunwen Lei · 2025
Read Generalization Bounds for Rank-sparse Neural Networks by Antoine Ledent, Rodrigo Alves, Yunwen Lei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21945.txt.
Introduction
Antoine Ledent, Rodrigo Alves, and Yunwen Lei ask why deep networks generalize instead of merely memorizing, focusing on a curious pattern. As networks deepen, their internal weight and activation tables settle into an approximately low-rank shape, as if the data forces a bottleneck.
Building on that observation, the trio proves learning guarantees that sharpen when the low-rank structure is present. The bounds run through a family of size measures for matrices, called Schatten quasi-norms, which stay small when a matrix is nearly low-rank. In plain terms, the number of examples needed grows with network width and depth but is discounted by the bottleneck rank. A network that compresses well needs less data to be trusted. The results connect a training habit, weight decay, to the same compression story.
Catalog record
| CREATOR | Antoine Ledent, Rodrigo Alves, Yunwen Lei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 207 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21945