Revisiting the Shape Convention of Transformer Language Models

Feng-Ting Liao, Meng-Hsi Chen, Guan-Ting Yi, Da-shan Shiu · 2026

Read Revisiting the Shape Convention of Transformer Language Models by Feng-Ting Liao, Meng-Hsi Chen, Guan-Ting Yi, Da-shan Shiu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06471.txt.

Description

The shape of a transformer has barely changed in years: feed-forward blocks swell wide, then narrow again. Feng-Ting Liao and 4 co-authors question that convention. Theory and tests suggest hourglass blocks, wide then narrow then wide, stay expressive despite their bottleneck, so the team built Hourglass Transformers around them.

Hourglass attention also separates the residual stream's width from attention width. Squeezing the feed-forward middle allows wider streams and fewer layers at the same parameter count. Across scales up to billions of parameters, the hourglass models matched conventional ones on language and downstream tasks, with better training efficiency at matched accuracy. After long-context extension, the largest model beat its baseline across context lengths, with cheaper attention at the longest settings.

Catalog record

CREATORFeng-Ting Liao, Meng-Hsi Chen, Guan-Ting Yi, Da-shan Shiu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-06471

Browse the full library · About the Tower of Babel