An overview of condensation phenomenon in deep learning
Zhi-Qin John Xu, Yaoyu Zhang, Zhangchen Zhou · 2025
Read An overview of condensation phenomenon in deep learning by Zhi-Qin John Xu, Yaoyu Zhang, Zhangchen Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09484.txt.
Introduction
Zhi-Qin John Xu, Yaoyu Zhang, and Zhangchen Zhou review a clustering effect in neural network training in this 2025 paper. During training, neurons in the same layer tend to gather into groups with like outputs. This effect is called condensation, and the number of groups usually grows as training goes on.
Small starting weights and a common training trick called dropout can both help this gathering along. The authors examine why it happens, from both the motion of training and the shape of the error surface. They argue the effect gives clues to why networks generalize well to new data. It also links to stronger reasoning in transformer language models, they note.
Catalog record
| CREATOR | Zhi-Qin John Xu, Yaoyu Zhang, Zhangchen Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-09484
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An overview of condensation phenomenon in deep learning Zhi-Qin John Xu, Yaoyu Zhang, Zhangchen Zhou Abstract In this paper, we provide an overview of a common phenomenon, condensation, observed during the nonlinear training of neural networks: During the nonlinear training of neural networks, neurons in the same layer tend to condense into groups with similar outputs. Empirical observations suggest that the number of condensed clusters of neurons in the same layer typically increases monotonically as training progresses. Neural networks with small weight initializations or Dropout optimization can facilitate this condensation process. We also examine the underlying mechanisms of condensation from the perspectives of training dynamics and the structure of the loss landscape. The condensation phenomenon offers valuable insights into the generalization abilities of neural networks and correlates to stronger reasoning abilities in transformer-based language models. 1 Introduction Deep neural networks (DNNs) have demonstrated remarkable performance across a wide range of applications. In particular, scaling laws suggest that improvements in performance for Large Language Models (LLMs) are closely tied to the size of both the model and the dataset [ 21 ] . Understanding how these large-scale neural networks achieve such extraordinary performance is crucial for developing principles that guide the design of more efficient, robust, and computationally cost-effective machine learning models. However, the study of large neural networks presents significant challenges, such as their enormous parameters and complex network architectures. Additionally, the data—ranging from language to image data—are often too complex to analyze using traditional methods. In this context, a phenomenon-driven approach has proven to be effective in uncovering insights into the behavior of neural networks. One such phenomenon is the over-parameterization puzzle, which has led to a deeper understanding of neural network generalization [ 5 , 48 ] . This puzzle reveals that a neural network can generalize well even when the number of parameters far exceeds the number of training data points. This observation challenges traditional learning theory, which typically improves generalization by imposing constraints on model complexity [ 41 ] . In contrast, the generalization of large neural networks appears to be largely independent of superficial complexity, such as the number of parameters. Instead, the optimization trajectory plays a crucial role in locating a minimum with specific properties among various types of minima. Empirical studies have shown that smaller batch sizes in Stochastic Gradient Descent (SGD) tend to lead to flatter minima, which is associated with better generalization [ 22 ] . This led to the development of sharpness-aware minimization (SAM) [ 16 ] techniques that further improve generalization performance. Additionally, recent works have shown that the noise covariance induced by SGD aligns with the Hessian of the loss landscape [ 59 , 43 , 15 ] , providing further insights into the optimization dynamics. Another important empirical finding is the existence of a simplicity bias during neural network training [ 1 ] . A series of experiments, followed by theoretical analysis, has identified a low-frequency bias, known as the frequency principle [ 44 , 45 ] or spectral bias [ 34 ] , which helps explain the observed differences in generalization performance. This principle has also inspired the development of multi-scale DNN architectures [ 26 , 25 , 7 ] and Fourier feature networks [ 40 ] , which accelerate the learning of high-frequency components in the data. To further investigate the simplicity bias, several studies have analyzed the evolution of neural network parameters during training. Two distinct regimes [ 28 , 56 ] have been identified: the linear regime, in which parameters initialized with relatively large values undergo minimal changes during training, and the nonlinear regime, where smaller initializations result in more substantial parameter adjustments [ 35 , 13 ] . In the linear regime, the behavior of the neural network closely resembles that of kernel methods, with the neural tangent kernel (NTK) [ 19 , 14 ] being a prominent example. The transition between the linear and nonlinear regimes represents a critical phase, with mean-field dynamics being a typical example [ 31 , 37 , 35 ] . It is in the nonlinear regime that a universal condensation phenomenon occurs [ 28 , 57 , 56 ] . In this paper, we aim to overview previous works on this phenomenon and provide a unified description of condensation: Condensation : During the nonlinear training of neural networks, neurons in the same layer tend to condense into groups with similar outputs. This condensed regime represents a state in which neurons in the same layer condense into a few distinct groups, with neurons within each group performing similar functions. This clustering phenomenon implies that a wide neural network can behave similarly to a much narrower network. Early in the nonlinear training process, neurons in the same layer tend to group into a small number of clusters [ 30 , 33 , 29 , 4 , 57 , 56 , 32 , 42 ] . As training progresses, the number of clusters increases, which facilitates fitting. Thus, the condensation phenomenon offers a mechanism for the increasing complexity of the network’s outputs as training progresses. In this paper, we present experiments with various neural network architectures to demonstrate the ubiquity of the condensation phenomenon in nonlinear training [ 27 , 57 , 58 ] . We also explore how dropout [ 38 ] implicitly induces a bias toward condensation [ 55 , 52 ] . Furthermore, we examine the origins of condensation from the perspectives of loss landscapes and training dynamics. The condensation phenomenon suggests a potential pruning strategy, where network size can be reduced without sacrificing generalization ability [ 50 , 8 ] . This insight also leads to a novel optimistic estimation of the sample size required to recover a target function based on a perfectly condensed network [ 51 ] rather than relying on superficial network complexity, where the latter often leads to overly conservative estimates. Moreover, the condensation phenomenon, originally observed in simple two-layer neural networks, provides a deeper understanding of the reasoning and memorization processes in transformer models, particularly for language tasks [ 53 , 54 ] . This understanding could also inform methods for training transformer networks with improved reasoning capabilities. Given that condensation is a prominent feature of the nonlinear training of neural networks, a deep understanding of this phenomenon would significantly enhance our comprehension and more effective utilization of deep learning. This phenomenon has been characterized in various ways throughout the literature. [ 30 ] described it as a quantization effect where weight vectors tend to concentrate in finite directions due to gradient descent. [ 6 ] referred to it as the weight clustering effect. [ 14 ] provided an illustrative example of non-lazy training. [ 33 ] named this behavior a form of inductive bias. Several works investigated this behavior of neurons within the same layer and named it “alignment/get align” [ 20 , 29 , 4 , 12 , 32 , 3 ] . [ 23 , 24 ] termed this phenomenon “directional convergence”. 2 Concept of condensation The concept of condensation refers to the tendency of neurons within the same layer to condense into groups with similar outputs during training. This alignment or clustering of neurons is influenced