The Implicit Bias of Depth: From Neural Collapse to Softmax Codes

Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis · 2026

Read The Implicit Bias of Depth: From Neural Collapse to Softmax Codes by Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23087.txt.

Introduction

Garrod, Keating, and Thrampoulidis study the implicit bias of depth in the deep free feature model, equal to a linear network with orthogonal inputs trained without rules. They show depth induces a low rank bias: low rank arrays spread norm well through chained products, pushing low rank alternatives to neural collapse. These correspond to softmax codes, max margin solutions found before in narrow nets. Early time push among single values drives low rank rise, and depth shrinks collapse's basin, while width biases at random started nets toward rank.

Catalog record

CREATORConnall Garrod, Jonathan P. Keating, Christos Thrampoulidis
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE136486
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23087

Browse the full library · About the Tower of Babel