A Two-Phase Perspective on Deep Learning Dynamics
Robert de Mello Koch, and Animik Ghosh · 2025
Read A Two-Phase Perspective on Deep Learning Dynamics by Robert de Mello Koch, and Animik Ghosh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12700.txt.
Introduction
Deep learning may proceed in two phases, Robert de Mello Koch and Animik Ghosh propose in a 2025 paper. The first phase is rapid curve fitting. The second is a slower compression, or coarse graining, phase. That view is supported by a shared time pattern in three known phenomena: grokking, double descent, and the information bottleneck, each with delayed generalization.
The authors show empirically that the timescales of those phenomena align in two rather different settings. Mutual information between hidden layers and input data serves as a natural progress measure. They argue the second phase is not actively optimized by standard training and may be prolonged without need. An analogy with the renormalization group frames the phase as principled forgetting.
Catalog record
| CREATOR | Robert de Mello Koch, and Animik Ghosh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-12700
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Two-Phase Perspective on Deep Learning Dynamics Robert de Mello Koch, and Animik Ghosh Abstract We propose that learning in deep neural networks proceeds in two phases: a rapid curve-fitting phase followed by a slower compression or coarse-graining phase. This view is supported by the shared temporal structure of three phenomena – grokking, double descent, and the information bottleneck – all of which exhibit a delayed onset of generalization well after training error reaches zero. We empirically show that the associated time scales align in two rather different settings. Mutual information between hidden layers and input data emerges as a natural progress measure, complementing circuit-based metrics such as local complexity and the linear mapping number. We argue that the second phase is not actively optimized by standard training algorithms and may be unnecessarily prolonged. Drawing on an analogy with the renormalization group, we suggest that this compression phase reflects a principled form of forgetting, critical for generalization. 1 Introduction Grokking is the phenomenon where a neural network, after memorizing its training data and showing poor generalization, suddenly and dramatically begins to generalize well – typically long after achieving zero training error Grok1 ; Grok2 ; Grok3 . Originally observed in synthetic tasks, recent studies suggest that grokking is surprisingly widespread and can arise in a range of practical settings AlwaysGrok . A compelling reason to study grokking is that it challenges classical learning theory. It exposes a mismatch between fitting and understanding: networks can memorize perfectly yet take much longer to internalize the correct algorithm or structure, as seen in tasks like modular arithmetic. This delayed transition reveals a richer set of dynamics underlying generalization than standard bias-variance tradeoffs capture AlwaysGrok ; Grok4 ; Grok5 ; Grok6 . Grokking also shows that training dynamics remain active well after the loss plateaus. Internal representations may continue evolving in subtle ways that ultimately enable generalization. Understanding these dynamics can shed light on hidden mechanisms governing learning, help us better navigate the optimization landscape, and inform practical decisions like early stopping, regularization, and learning rate schedules. From a research perspective, grokking provides a tractable, interpretable setting – what one might call the “fruit fly” of generalization. It allows us to study deep learning dynamics in a clean environment where signals like compression, abstraction, and memorization can be precisely tracked. In this paper, we argue that grokking is a manifestation of a deeper structure: that learning proceeds as a two-stage process – first through curve fitting, and later through a phase resembling coarse graining or “forgetting”. We make this argument by identifying a common structure between three phenomena: grokking, double descent, and the information bottleneck principle. To set the stage, we briefly review the relevant aspects of double descent and the information bottleneck. Double descent refines the classical view of generalization by showing that test error, after initially rising near the interpolation threshold, can decrease again as model complexity increases. This non-monotonic behavior has been observed across a wide range of models, including neural networks, kernel methods, and even simple linear regression. The information bottleneck, meanwhile, offers a principled framework for understanding how a model balances compression and predictive power. It posits that the optimal internal representation of data is one that retains as much relevant information about the target as possible, while discarding irrelevant input features. We propose that these frameworks are not merely analogous but reflect a common underlying structure, inducing the same fundamental phases in the learning process. Specifically, we demonstrate that three critical time scales—the point at which grokking achieves perfect training accuracy, the onset of the second descent in double descent, and the transition to compression in the information bottleneck—are empirically aligned. In Section 2 , we define these time scales, explain their significance, and outline how they can be measured in practice. Section 3 presents the empirical evidence supporting this alignment. In Section 4 , we motivate the hypothesis that learning unfolds in two distinct stages: an initial curve-fitting phase, followed by a second phase resembling coarse graining or “forgetting”. This picture is consistent with detailed numerical studies of grokking. A key outcome of our work is the proposal that the mutual information between inputs and outputs of a hidden layer serves as a natural progress measure for deep network training. Finally, in Section 5 , we discuss the implications of our findings and suggest directions for future work. 2 Time Scales and their relevance Grokking offers a tractable and interpretable setting for studying generalization in deep neural networks. The insights it provides directly motivate the central hypothesis we present in Section 4 . To build toward that, we review key supporting results in this section. Section 2.1 revisits the main observations about grokking, which already hint that learning unfolds in two distinct phases. To identify the first of these phases, we turn to the phenomenon of double descent – an update to the classical bias-variance trade-off – which we review in Section 2.2 . The second phase, characterized by a form of compression, is naturally described by the information bottleneck framework, reviewed in Section 2.3 . 2.1 Grokking Grokking refers to the phenomenon in which a deep neural network (DNN) begins to generalize well only after a prolonged period of training, long after achieving near-zero training error. By now, several detailed dynamical studies have explored grokking, offering an appealing and intuitive understanding of some of its underlying mechanisms Grok1 ; explgrok1 ; Grok4 . A key concept we will build on is that of a circuit explgrok2 . To motivate this idea, we begin by recalling that a DNN applies a sequence of nonlinear transformations to an input vector \vec{x} , propagating it through L layers. This defines a function f_{\theta}(\vec{x}) of the form f_{\theta}(\vec{x})\equiv a\left(W^{(L)}a\Bigg(\cdots a\Big(W^{(2)}a(W^{(1)}\vec{x}+b^{(1)})+b^{(2)}\Big)\cdots+b^{(L-1)}\Bigg)+b^{(L)}\right) (1) where \theta denotes the complete set of network parameters. Each layer l contributes a weight matrix W^{(l)}_{ij} and a bias vector b^{(l)}_{i} to \theta . The activation function a(\cdot) is a nonlinearity applied element-wise to the output of each neuron. In our case, we use the ReLU activation function, a(t)=\max(0,t), (2) which outputs zero whenever t<0 . This implies that such neurons are effectively inactive and do not contribute to the network’s output. As a result, for any given input, the computation performed by the network involves only a subset of neurons and connections. Figure 1: Circuits in a ReLU MLP (A) Schematic of a multi-layer perceptron (MLP) with an input and output layer, and four hidden layers. (B) A representative activation pattern in which only the top neuron in the second hidden layer and the top two neurons in the third hidden layer are active. The resulting circuit includes only the activated neurons and the weights connected between them. (C) Transition to a neighboring linear region corresponds to the addition or removal of a single neuron from the circuit. In the example shown, the second neuron in the second hidden layer becomes newly activated. This idea can be formalized using the notion of a circuit , as illustrated in Figure 1 . A circuit consists of the subset of neurons that are activated for a given input, along with the weights that connect them. For that input, the