Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
Vatsal Baherwani, Zixi Chen, Shikai Qiu, Andrew Gordon Wilson, Pavel Izmailov · 2026
Read Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns by Vatsal Baherwani, Zixi Chen, Shikai Qiu, Andrew Gordon Wilson, Pavel Izmailov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25010.txt.
Description
Vatsal Baherwani, Zixi Chen, and three colleagues explain why capabilities in language models appear suddenly during training. Scaling laws predict smooth loss improvements, yet skills like in context learning switch on abruptly past some scale. Their experiments show emergence is random: capabilities arise at random points through training, with larger models acquiring them earlier. The trigger is attention. Skills such as pattern completion and indirect object identification switch on when the model abruptly learns the task relevant sparse attention pattern.
Synthetic tasks, linear maps and cellular automata, isolate the effect. Learning such a pattern grows harder with longer contexts and sparser patterns. Adding heads improves learning speed. Widening each head gives diminishing returns past a minimum capacity. Architectures differ: an MLP Mixer beats a transformer on linear map tasks needing complex patterns. Emergence, they conclude, is the signature of hard to learn attention patterns clicking into place.
Catalog record
| CREATOR | Vatsal Baherwani, Zixi Chen, Shikai Qiu, Andrew Gordon Wilson, Pavel Izmailov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25010