Generalization Bounds for Rank-sparse Neural Networks

Antoine Ledent, Rodrigo Alves, Yunwen Lei · 2025

Read Generalization Bounds for Rank-sparse Neural Networks by Antoine Ledent, Rodrigo Alves, Yunwen Lei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21945.txt.

Introduction

Antoine Ledent, Rodrigo Alves, and Yunwen Lei ask why deep networks generalize instead of merely memorizing, focusing on a curious pattern. As networks deepen, their internal weight and activation tables settle into an approximately low-rank shape, as if the data forces a bottleneck.

Building on that observation, the trio proves learning guarantees that sharpen when the low-rank structure is present. The bounds run through a family of size measures for matrices, called Schatten quasi-norms, which stay small when a matrix is nearly low-rank. In plain terms, the number of examples needed grows with network width and depth but is discounted by the bottleneck rank. A network that compresses well needs less data to be trusted. The results connect a training habit, weight decay, to the same compression story.

Catalog record

CREATORAntoine Ledent, Rodrigo Alves, Yunwen Lei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE207 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21945

Browse the full library · About the Tower of Babel