Glassy dynamics near the interpolation transition in deep recurrent networks
John Hertz, Joanna Tyrcha · 2024
Read Glassy dynamics near the interpolation transition in deep recurrent networks by John Hertz, Joanna Tyrcha (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10094.txt.
Introduction
John Hertz and Joanna Tyrcha study in 2024 slow, glass like learn acts in deep loop nets near the point where train fit turns. That point splits nets that can just fit the data from nets with room to spare. Their train set is four part choir works in an old style.
From the spare side, learn times swell as width nears the split line, in a clean math form. Past waits also slow the net's jiggle in the spare and split zones, as in glass. In the tight zone that jiggle law holds in a pure form. Loss caps from the step rule shape the long tail.
Catalog record
| CREATOR | John Hertz, Joanna Tyrcha |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10094
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Glassy dynamics near the interpolation transition in deep recurrent networks John Hertz, Joanna Tyrcha Abstract We examine learning dynamics in deep recurrent networks, focusing on the behavior near the boundary in the depth-width plane separating under- from overparametrized networks, known as the interpolation transition. The training data are Bach chorales in 4-part harmony, and the learning is by stochastic gradient descent with a cross-entropy loss function. We find critical slowing down of the learning approaching the transition from the overparametrized side: For a given network depth, learning times to reach small training loss values appear to diverge proportional to 1 / ( w − w c ) 1/(w-w_{c}) as the width w w approaches a (loss-dependent) critical value w c w_{c} . We identify the zero-loss limit of this value with the interpolation transition. We also study aging (the slowing down of fluctuations as the time since the beginning of learning increases). Taking a system that has been learning for a time τ w \tau_{w} , we measure the subsequent mean-square fluctuations of the weight values at times τ > τ w \tau>\tau_{w} . In the underparametrized phase, we find that they are well-described by a single function of τ / τ w \tau/\tau_{w} . While this scaling holds approximately at short times at the transition and in the overparametrized phase, it breaks down at longer times when the training loss gets close to the lower limit imposed by the stochastic gradient descent dynamics. Both this kind of aging and the critical slowing down are also found in certain spin glass models, suggesting that those models contain the most essential features of the learning dynamics. I Introduction Recurrent neural networks are an attractive tool for modeling dynamical systems based on time-series data. They contain implicitly a mechanism for storing and using information about past states of the system to generate the next state. This property is essential, for example, in applications to music generation, where information about the distant past can be necessary to generate the correct temporal structure in the near future. In understanding any learning network, an essential feature is what has come to be called the interpolation threshold: the boundary in hyperparameter space separating regions where a training set can be learned (i.e., the network can successfully interpolate between all the training points) from regions where it can’t be. Some work [ 1 , 2 ] has explored aspects of this problem for standard layered networks, exploiting an analogy with jamming transitions [ 3 ] . More recently, Kerr Winter and Janssen [ 4 ] explored learning dynamics in a variety of such networks, including effects of regularization and Langevin training noise. In some cases they found results qualitatively similar to ours, as we will discuss later. Of particular relevance to our work is that of Baity-Jesi et al [ 5 ] . Although they did not study the transition itself, they performed interesting learning experiments for both strongly underparametrized and strongly overparametrized (non-recurrent) networks. They found spin-glass-like aging [ 6 , 7 , 8 ] in a strongly underparametrized network, as well as something similar, persisting for rather long times, in a number of strongly overparametrized networks. However this apparent aging always came to an end as the networks approached successful learning, and the asymptotic dynamics were simple relaxation. In our work we extend this kind of study to recurrent networks, with special attention to the dynamics near the transition between underparametrized and overparameterized networks. We choose to do this within an otherwise conventional layered network architecture trained by stochastic gradient descent (SGD), just adding full recurrence within each layer. There are of course many possible recurrent architectures, but this seemed to be the simplest extension of the standard model network to include recurrence. Our aim is to gain an understanding of the asymptotic learning dynamics in both underparametrized and overparametrized networks near the transition. In doing so, like Bati-Jesi et al , we will use ideas from spin glass theory, where related transitions are well-studied. The defining characteristic of a spin glass system is the existence of “glassy” states where the degrees of freedom (called “spins”) become frozen or nearly so. The key features of spin-glass systems that are relevant to our work are the following: (1) Quite generally, they have phase transitions, driven by changes in temperature, external fields or other control parameters, between “normal” and glassy phases. In some cases, the approach to the transition from the normal side can exhibit “critical slowing down”, meaning that the relaxation times of the system diverge as one approaches the transition. (2) The dynamical signature of the glassy phase is aging . The basic idea of aging is this: If a system is suddenly quenched into a glassy state, it will gradually relax and approach but never reach thermal equilibrium. If we study a system starting at a time \tau_{w} after the quench, the properties we find will depend on \tau_{w} : the “older” the system (i.e., the larger \tau_{w} is), the slower the dynamics we will observe. In soluble model systems [ 9 ] , this can be expressed simply: the correlation between degrees of freedom at \tau_{w} and at \tau_{w}+\tau , with \tau of order \tau_{w} , will depend only on the ratio \tau/\tau_{w} , reflecting the fact that \tau_{w} is the only relevant characteristic time in the system. Further details of the spin glass models relevant to our work can be found in Appendix A. In the present work, we have trained recurrent nets to imitate Bach’s chorales (sequences of chords in 4-part harmony) and, like Baity-Jesi et al , asked whether and to what degree they exhibited dynamical properties like spin glass systems. We have studied the phase diagram of these networks in the space defined by their width w and depth d and investigated the neighborhood of the interpolation transition as follows: For each depth considered, we find the smallest width, w_{c}(l) , for which the loss can be trained down to a specific small target value l . The values of w_{c}(l) for different l form a line in the w - l plane. We find that as one approaches this line from the w>w_{c}(l) side, critical slowing down occurs. The learning time to reach w_{c}(l) diverges, and for w<w_{c}(l) the loss can never be brought below l . We identify the zero- l limit of this line with the interpolation transition. We call the region of the w - l plane to the left of this line underparametrized and the region to the right overparametrized. In the underparametrized region, we find aging like that in a class of mean-field spin-glass models [ 6 , 7 , 8 ] . In the overparametrized region, this aging disappears as the learning approaches its final steady state. The paper is organized as follows. In Sec. II we describe the model networks we study, the training data, and how they are trained. In Sec. III we present general features of the training results: the shapes of the learning curves for under-, over- and critically parametrized networks, showing how they depend on network depth. We also examine the asymptotic long-time decay of the loss