Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns

Vatsal Baherwani, Zixi Chen, Shikai Qiu, Andrew Gordon Wilson, Pavel Izmailov · 2026

Read Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns by Vatsal Baherwani, Zixi Chen, Shikai Qiu, Andrew Gordon Wilson, Pavel Izmailov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25010.txt.

Description

Vatsal Baherwani, Zixi Chen, and three colleagues explain why capabilities in language models appear suddenly during training. Scaling laws predict smooth loss improvements, yet skills like in context learning switch on abruptly past some scale. Their experiments show emergence is random: capabilities arise at random points through training, with larger models acquiring them earlier. The trigger is attention. Skills such as pattern completion and indirect object identification switch on when the model abruptly learns the task relevant sparse attention pattern.

Synthetic tasks, linear maps and cellular automata, isolate the effect. Learning such a pattern grows harder with longer contexts and sparser patterns. Adding heads improves learning speed. Widening each head gives diminishing returns past a minimum capacity. Architectures differ: an MLP Mixer beats a transformer on linear map tasks needing complex patterns. Emergence, they conclude, is the signature of hard to learn attention patterns clicking into place.

Catalog record

CREATORVatsal Baherwani, Zixi Chen, Shikai Qiu, Andrew Gordon Wilson, Pavel Izmailov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25010

Browse the full library · About the Tower of Babel