Tapered Language Models
Reza Bayat, Ali Behrouz, Aaron Courville · 2026
Read Tapered Language Models by Reza Bayat, Ali Behrouz, Aaron Courville (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23670.txt.
Description
Bayat, Behrouz, and Courville argue that language models should be built wider at the front and narrower at the back, in a 2026 paper on neural network design. Nearly all language models stack identical layers, spreading parameters evenly across depth, a default inherited from the original transformer. Evidence suggests later layers mostly refine an almost finished answer rather than transforming it. If layers contribute unevenly, capacity should too.
A controlled experiment confirms the hunch: under a fixed budget, giving earlier layers more capacity and later layers less improves perplexity, the standard score of prediction quality, while the reverse hurts. The authors define tapered language models, where a component shrinks steadily with depth. They taper the feed forward blocks, which hold most parameters, on a smooth cosine schedule. Across three scales and four architectures, tapering improves both perplexity and downstream benchmark scores, at no extra parameter or compute cost.
Catalog record
| CREATOR | Reza Bayat, Ali Behrouz, Aaron Courville |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.23670