Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions

Wei Wenbo, 1 Nicholas Chong Jia Le, 1 Lai Choy Heng, 1 and Feng Ling 2,1 · 2025

Read Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions by Wei Wenbo, 1 Nicholas Chong Jia Le, 1 Lai Choy Heng, 1 and Feng Ling 2,1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20030.txt.

Introduction

Wenbo Wei and co-authors report a new pattern in deep learning in a paper from two thousand twenty-five. While training a recurrent network called LSTM on a real task, they saw performance rise and fall in long cycles. The cycles continued well after the point where training is usually stopped.

Stability analysis tied the cycles to shifts between order and chaos in the model's dynamics. The best training steps sat right at the edge between the two phases. The single best point came at the first shift from order to chaos, where that edge zone was widest. A wide edge, they argue, lets the network search weight settings best. The finding suggests training length should watch phase shifts, not just loss.

Catalog record

CREATORWei Wenbo, 1 Nicholas Chong Jia Le, 1 Lai Choy Heng, 1 and Feng Ling 2,1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20030

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions Wei Wenbo, 1 Nicholas Chong Jia Le, 1 Lai Choy Heng, 1 and Feng Ling 2,1 Abstract We observe a novel ‘multiple-descent’ phenomenon during the training process of LSTM, in which the test loss goes through long cycles of up and down trend multiple times after the model is overtrained. By carrying out asymptotic stability analysis of the models, we found that the cycles in test loss are closely associated with the phase transition process between order and chaos, and the local optimal epochs are consistently at the critical transition point between the two phases. More importantly, the global optimal epoch occurs at the first transition from order to chaos, where the ‘width’ of the ‘edge of chaos’ is the widest, allowing the best exploration of better weight configurations for learning. I Introduction In deep learning, understanding the training dynamics has become paramount for enhancing model performance, generalization, and robustness. The training of deep neural networks involves navigating through complex, high-dimensional parameter spaces, where the interplay between model complexity, dataset characteristics, and learning algorithms dictates the learning trajectory. This process is far from straightforward, often characterized by phenomena such as overfitting, underfitting, and various forms of descent in performance metrics. The dynamics of training deep neural networks are critical for several reasons. Generalization is a primary concern in machine learning, focusing on the model’s ability to generalize from training data to unseen data. Understanding how training dynamics affect generalization can lead to better strategies for model tuning ( Geiger et al. 2020 ; Zhang et al. 2021 ) . Optimization within deep networks is challenging due to the non-convex nature of loss landscapes, where insights into training dynamics inform the choice of optimizers, learning rates, and regularization techniques ( Li et al. 2018 ; Reddi et al. 2018 ) . Overfitting and underfitting are pivotal issues; the former occurs when a model memorizes training data too closely, while the latter happens when it fails to capture underlying patterns. Studying training dynamics aids in determining when to stop training to avoid these pitfalls ( Srivastava et al. 2014 ; Goodfellow et al. 2016 ) . Lastly, dissecting the training process can enhance model interpretability by revealing which network parts are crucial for learning ( Zeiler and Fergus 2014 ; Olah et al. 2017 ) . The current understanding of training dynamics in deep learning involves several key concepts. The concept of the edge of stability posits that networks perform best when their parameters balance between order and chaos, optimizing learning capacity ( Cohen et al. 2021 ; Arora et al. 2018 ) . The Neural Tangent Kernel (NTK) framework offers insights into how wide neural networks learn by resembling kernel methods in the infinite-width limit ( Jacot et al. 2018 ; Lee et al. 2019 ) . Implicit regularization by optimization algorithms leads to better-than-expected generalization ( Neyshabur et al. 2017 ; Gunasekar et al. 2018 ) . Exploring the loss landscape provides a visual understanding of how different optimization paths lead to minima with varied generalization capabilities ( Li et al. 2018 ; Garipov et al. 2018 ) . The scale of gradient noise during training correlates with model generalization, with higher noise often linked to better generalization ( Smith et al. 2018 ; Keskar et al. 2017 ) . Lastly, normalization techniques like batch normalization have shown to stabilize learning and improve generalization ( Ioffe and Szegedy 2015 ; Santurkar et al. 2018 ) . The double descent phenomenon has emerged as a significant area of study in recent years, particularly in the context of deep learning. Initially observed in classical statistics ( Belkin et al. 2019 ) , this phenomenon describes a U-shaped curve in model performance where, after an initial decrease, the test error increases with model complexity or training duration before decreasing again. This behavior challenges the traditional understanding of the bias-variance trade-off and has been both empirically observed and theoretically analyzed in the context of neural networks ( Nakkiran et al. 2021 ; Mei and Montanari 2022 ) . However, while double descent provides insights into how model complexity affects generalization, several open questions remain. These include the exact mechanisms behind why increasing model capacity can lead to better performance after an initial decline, the role of dataset size and noise, and how these observations extend to different types of architectures or learning tasks. Our study introduces a novel observation of ‘multiple descents’ in the training of deep learning models, specifically in Long Short-Term Memory (LSTM) networks. We explore this through an extensive analysis of LSTM networks trained on the Large Movie Review Dataset for sentiment analysis. Our research reveals that during the training process, particularly in the overfitting phase, the test loss exhibits multiple cycles of increases followed by a sharp decline, a pattern not fully captured by existing models like double descent. And more importantly, the most optimal epoch is consistently at the first transition from order to chaos, where the ‘edge of chaos’ is the widest, allowing the best exploration of better weight configurations for learning. One core contribution of this study is the identification of these cycles with specific phase transitions. We propose that each descent in test loss corresponds to a transition between order and chaos state during the evolution of the neural network from training, and the optimal performance is achieved at the edge of these transitions, particularly at the first such transition. By employing techniques from dynamical systems, specifically asymptotic stability analysis, we measure how perturbations in initial conditions propagate through the network, offering insights into when these order-chaos transitions occur. We will detail the experimental setup where we over-train an LSTM model to observe these dynamics, provide some theoretical explanation why these transitions might occur, by drawing parallels with known behaviors in non-linear systems like the \tanh map, and discuss the implications of these findings for training deep learning models, particularly in terms of finding optimal training epochs. II Experiment Setup We build a basic LSTM model to perform sentiment analysis of the Large Movie Review Dataset Maas et al. 2011 , which contains 50,000 labelled movie reviews from the Internet Movie Database (IMDb) and is a popular open-source dataset for natural language processing. The model is over-trained to 10000 epochs to induce overfitting in order to explore the rich order chaos transition behaviors. The embedding layer projects the individual words (tokens) into a learned continuous vector space, the LSTM layer performs the time-series analysis, and the output layer performs the final classification. The hyperparameters of the model are summarized in Table 1 . We partition 70% of the dataset for training and 30% for testing, padding/truncating each review to a fixed length of 500 tokens. The model is optimized with the Adam optimizer with a learning rate of 0.0005. Table 1: Structure of the model Layer Output Dimension Embedding Layer 32 LSTM Layer 60 Fully Connected Layer 1 II.1 Asymptotic stability analysis We calculate the asymptotic trajectory separation/difference of the output recurrent unit of LSTM cell h_{t} under perturbation to extract the asymptotic stability of the LSTM model, specifically order and chaos phases. The key

Read the complete text