Tapered Language Models

Reza Bayat, Ali Behrouz, Aaron Courville · 2026

Read Tapered Language Models by Reza Bayat, Ali Behrouz, Aaron Courville (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23670.txt.

Description

Bayat, Behrouz, and Courville argue that language models should be built wider at the front and narrower at the back, in a 2026 paper on neural network design. Nearly all language models stack identical layers, spreading parameters evenly across depth, a default inherited from the original transformer. Evidence suggests later layers mostly refine an almost finished answer rather than transforming it. If layers contribute unevenly, capacity should too.

A controlled experiment confirms the hunch: under a fixed budget, giving earlier layers more capacity and later layers less improves perplexity, the standard score of prediction quality, while the reverse hurts. The authors define tapered language models, where a component shrinks steadily with depth. They taper the feed forward blocks, which hold most parameters, on a smooth cosine schedule. Across three scales and four architectures, tapering improves both perplexity and downstream benchmark scores, at no extra parameter or compute cost.

Catalog record

CREATORReza Bayat, Ali Behrouz, Aaron Courville
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.23670

Browse the full library · About the Tower of Babel