A dynamic view of some anomalous phenomena in SGD

Vivek S. Borkar · 2025

Read A dynamic view of some anomalous phenomena in SGD by Vivek S. Borkar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01751.txt.

Description

Vivek Borkar offers one dynamical story that explains two of deep learning's strangest training curves. His 2025 note addresses double descent, where test error falls, rises, and falls again as models grow or training proceeds, and grokking, where a network memorizes for ages before suddenly generalizing. Both look like magic from the loss curve alone.

His lens is two-timescale stochastic approximation, applied to the continuous-time limit of gradient dynamics. When parts of a system learn at different speeds, their interaction can produce plateaus, reversals, and delayed transitions that mimic the anomalies. The note is an explanation, not an algorithm: it proposes a mechanism consistent with the observed shapes. Such accounts matter because practitioners currently treat these curves as folklore. A dynamical theory, if it holds, converts surprise into prediction, telling trainers when a plateau is a dead end and when it is a held breath.

Catalog record

CREATORVivek S. Borkar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01751

Browse the full library · About the Tower of Babel