Revisiting the Shape Convention of Transformer Language Models
Feng-Ting Liao, Meng-Hsi Chen, Guan-Ting Yi, Da-shan Shiu · 2026
Read Revisiting the Shape Convention of Transformer Language Models by Feng-Ting Liao, Meng-Hsi Chen, Guan-Ting Yi, Da-shan Shiu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06471.txt.
Description
The shape of a transformer has barely changed in years: feed-forward blocks swell wide, then narrow again. Feng-Ting Liao and 4 co-authors question that convention. Theory and tests suggest hourglass blocks, wide then narrow then wide, stay expressive despite their bottleneck, so the team built Hourglass Transformers around them.
Hourglass attention also separates the residual stream's width from attention width. Squeezing the feed-forward middle allows wider streams and fewer layers at the same parameter count. Across scales up to billions of parameters, the hourglass models matched conventional ones on language and downstream tasks, with better training efficiency at matched accuracy. After long-context extension, the largest model beat its baseline across context lengths, with cheaper attention at the longest settings.
Catalog record
| CREATOR | Feng-Ting Liao, Meng-Hsi Chen, Guan-Ting Yi, Da-shan Shiu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-06471