Inverse Depth Scaling From Most Layers Being Similar
Yizhou Liu, Sara Kangaslahti, Ziming Liu, Jeff Gore · 2026
Read Inverse Depth Scaling From Most Layers Being Similar by Yizhou Liu, Sara Kangaslahti, Ziming Liu, Jeff Gore (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05970.txt.
Description
Neural scaling laws link a language model's loss to its size, but depth and width may not pull in the same way. Yizhou Liu and three co-authors measure how depth alone affects loss, using large language models and small toy residual nets. They find that deeper models lose less, with loss shrinking roughly in inverse proportion to depth. The pattern shows up in both the big models and the toy nets.
The likely cause is not deeper step-by-step reasoning. Layers that act much alike seem to cut error by averaging, like a panel of similar judges, rather than by building richer chains of thought. That regime is wasteful yet sturdy, and may stem from how residual nets are built. The authors argue that faster models will need designs that push depth to do real compositional work.
Catalog record
| CREATOR | Yizhou Liu, Sara Kangaslahti, Ziming Liu, Jeff Gore |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-05970