The two clocks and the innovation window: When and how generative models learn rules

Binxu Wang, Emma Lucia Byrnes Finn, Bingbin Liu, [email protected], [email protected] · 2026

Read The two clocks and the innovation window: When and how generative models learn rules by Binxu Wang, Emma Lucia Byrnes Finn, Bingbin Liu, [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10019.txt.

Description

Wang and two colleagues studied when and how generative models learn rules, tracing two training time points called clocks. The first clock, tau-rule, marks when outputs first turn rule-valid. The second clock, tau-mem, marks when models start copying training data. Generative models trained on finite data face a tension: their training goal settles on the seen data spread, not the true spread they seek to learn.

Starting with parity and other binary rules and puzzles, the paper maps how the clocks depend on the learning setup. Tau-rule rises with rule hardness and falls with model size. Tau-mem stays nearly the same across rules and scales nearly with dataset size. The innovation window between the clocks widens with dataset size, narrows with hardness, and can vanish when tau-rule catches tau-mem. The same structure appears in DiT and GPT models, with design-dependent offsets.

Catalog record

CREATORBinxu Wang, Emma Lucia Byrnes Finn, Bingbin Liu, [email protected], [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10019

Browse the full library · About the Tower of Babel