Why and When Deep is Better than Shallow: Implementation-Agnostic State-Transition Model of Deep Learning
Sho Sonoda, Yuka Hashimoto, Isao Ishikawa, Masahiro Ikeda · 2025
Read Why and When Deep is Better than Shallow: Implementation-Agnostic State-Transition Model of Deep Learning by Sho Sonoda, Yuka Hashimoto, Isao Ishikawa, Masahiro Ikeda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15064.txt.
Description
Sho Sonoda, Yuka Hashimoto, Isao Ishikawa, and Masahiro Ikeda present a 2025 theory study of when deeper neural networks learn better than shallow ones. Their model treats each hidden layer as a map on a state space and stacks up to k of them before an output layer. Depth is compared fairly: deep and shallow nets share the same layer building blocks. The key cost of depth, they show, is the size of the set of all stacked maps, measured by metric entropy.
Under smooth layers, that cost grows at most polynomially with depth. Under shrinking or special layers, it stays flat or grows only like a logarithm. Balanced against fitting error, the theory yields a best depth that grows with sample size. Worked cases include deep ReLU networks and chain-of-thought computation. Every claim was machine-checked in the Lean proof system.
Catalog record
| CREATOR | Sho Sonoda, Yuka Hashimoto, Isao Ishikawa, Masahiro Ikeda |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 217 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15064