Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
Jinchang Zhu, Jindong Li, Yuwen Hao, Chengyu Zou, Rong Fu, Menglin Yang · 2026
Read Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining by Jinchang Zhu, Jindong Li, Yuwen Hao, Chengyu Zou, Rong Fu, Menglin Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10504.txt.
Description
Jinchang Zhu and five colleagues identify a failure mode in GPT-style pretraining. A causal-decoder block is layered: lower layers build the residual basis for upper layers. Upper layers can lock onto sharp attention patterns while the lower-layer features they watch are still settling. The authors call this premature upper-layer attention specialization. Their 2026 paper shows the failure is real.
Briefly slowing just the upper layers' query and key projections early in training lifts a 270M GPT-style decoder's perplexity, token efficiency, and downstream accuracy. Probes confirm the intervention leaves lower-layer routing on schedule; it stops upper attention from settling onto an immature residual basis. Matched ablations point to multiplicative gated feed-forward networks, not RMSNorm or bias removal, as the part holding back the upstream residual writes behind the failure in LLaMA-style blocks. A pathwise analysis unifies both findings under a bound.
Catalog record
| CREATOR | Jinchang Zhu, Jindong Li, Yuwen Hao, Chengyu Zou, Rong Fu, Menglin Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10504