Superposition unifies power-law training dynamics

Zixin Jessie Chen, Hao Chen, Yizhou Liu, Jeff Gore · 2026

Read Superposition unifies power-law training dynamics by Zixin Jessie Chen, Hao Chen, Yizhou Liu, Jeff Gore (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01045.txt.

Description

Zixin Jessie Chen and co-authors test how packed features shape the speed law of training in a teacher-pupil setup. With no packing, their math ties the speed exponent to data shape and channel weight. Once a packing bottleneck bites, the law flips to a plain one-over-time curve near 1, whatever the data or channel mix. That packed regime trains up to ten times faster than step-by-step learning unpacked. Since many big nets pack features, the find may bear on a wide swath of real models.

Catalog record

CREATORZixin Jessie Chen, Hao Chen, Yizhou Liu, Jeff Gore
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48752
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01045

Browse the full library · About the Tower of Babel