A dynamic view of some anomalous phenomena in SGD
Vivek S. Borkar · 2025
Read A dynamic view of some anomalous phenomena in SGD by Vivek S. Borkar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01751.txt.
Description
Vivek Borkar offers one dynamical story that explains two of deep learning's strangest training curves. His 2025 note addresses double descent, where test error falls, rises, and falls again as models grow or training proceeds, and grokking, where a network memorizes for ages before suddenly generalizing. Both look like magic from the loss curve alone.
His lens is two-timescale stochastic approximation, applied to the continuous-time limit of gradient dynamics. When parts of a system learn at different speeds, their interaction can produce plateaus, reversals, and delayed transitions that mimic the anomalies. The note is an explanation, not an algorithm: it proposes a mechanism consistent with the observed shapes. Such accounts matter because practitioners currently treat these curves as folklore. A dynamical theory, if it holds, converts surprise into prediction, telling trainers when a plateau is a dead end and when it is a held breath.
Catalog record
| CREATOR | Vivek S. Borkar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01751