Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks
Lorenzo Livi · 2026
Read Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks by Lorenzo Livi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29519.txt.
Introduction
Livi studies why recurrent neural networks struggle to learn long-range dependencies, links between inputs far apart in time. Under stochastic gradient descent, a past input's influence fades with the lag. If that fade is exponential, the data needed grows exponentially too. A power-law fade keeps the cost manageable.
Livi shows the fade's decay class is not fixed by the architecture. It emerges from how state and parameter dynamics couple during training. Two regimes appear: a collapsed regime with fast exponential forgetting, and an anti-collapsed regime with slow power-law forgetting. Training pulls time scales short, while rare heavy-tailed fluctuations push a few of them very long. The extended regime survives only when those pushes balance the pull. A single spectral exponent then governs the spread of time scales and, with it, the forgetting rate.
Catalog record
| CREATOR | Lorenzo Livi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 207 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29519