The Implicit Bias of Depth: From Neural Collapse to Softmax Codes
Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis · 2026
Read The Implicit Bias of Depth: From Neural Collapse to Softmax Codes by Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23087.txt.
Introduction
Garrod, Keating, and Thrampoulidis study the implicit bias of depth in the deep free feature model, equal to a linear network with orthogonal inputs trained without rules. They show depth induces a low rank bias: low rank arrays spread norm well through chained products, pushing low rank alternatives to neural collapse. These correspond to softmax codes, max margin solutions found before in narrow nets. Early time push among single values drives low rank rise, and depth shrinks collapse's basin, while width biases at random started nets toward rank.
Catalog record
| CREATOR | Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 136486 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23087