The two clocks and the innovation window: When and how generative models learn rules
Binxu Wang, Emma Lucia Byrnes Finn, Bingbin Liu, [email protected], [email protected] · 2026
Read The two clocks and the innovation window: When and how generative models learn rules by Binxu Wang, Emma Lucia Byrnes Finn, Bingbin Liu, [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10019.txt.
Description
Wang and two colleagues studied when and how generative models learn rules, tracing two training time points called clocks. The first clock, tau-rule, marks when outputs first turn rule-valid. The second clock, tau-mem, marks when models start copying training data. Generative models trained on finite data face a tension: their training goal settles on the seen data spread, not the true spread they seek to learn.
Starting with parity and other binary rules and puzzles, the paper maps how the clocks depend on the learning setup. Tau-rule rises with rule hardness and falls with model size. Tau-mem stays nearly the same across rules and scales nearly with dataset size. The innovation window between the clocks widens with dataset size, narrows with hardness, and can vanish when tau-rule catches tau-mem. The same structure appears in DiT and GPT models, with design-dependent offsets.
Catalog record
| CREATOR | Binxu Wang, Emma Lucia Byrnes Finn, Bingbin Liu, [email protected], [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 103 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10019