Neural Collapse beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime

Diyuan Wu, Marco Mondelli · 2025

Read Neural Collapse beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime by Diyuan Wu, Marco Mondelli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19104.txt.

Introduction

Diyuan Wu and Marco Mondelli wrote this 2025 computing paper on neural collapse, a pattern seen in well-trained networks. In collapse, the last-layer features of each class shrink toward a single point, a trait labeled NC1. Past proofs relied on a model that treats features as free variables, ignoring the data itself.

The authors study a three-layer network whose first layers learn in the mean-field regime. They prove that points with small loss and small gradient must nearly satisfy NC1, with the gap set by those two quantities. Gradient flow on squared error then flows to such collapse points. For well-separated data, collapse and near-zero test error arrive together. The results ground a striking pattern in the shape of the loss landscape itself.

Catalog record

CREATORDiyuan Wu, Marco Mondelli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE117 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-19104

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neural Collapse beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime Diyuan Wu, Marco Mondelli Abstract Neural Collapse is a phenomenon where the last-layer representations of a well-trained neural network converge to a highly structured geometry. In this paper, we focus on its first (and most basic) property, known as NC1: the within-class variability vanishes. While prior theoretical studies establish the occurrence of NC1 via the data-agnostic unconstrained features model, our work adopts a data-specific perspective, analyzing NC1 in a three-layer neural network, with the first two layers operating in the mean-field regime and followed by a linear layer. In particular, we establish a fundamental connection between NC1 and the loss landscape: we prove that points with small empirical loss and gradient norm (thus, close to being stationary) approximately satisfy NC1, and the closeness to NC1 is controlled by the residual loss and gradient norm. We then show that (i) gradient flow on the mean squared error converges to NC1 solutions with small empirical loss, and (ii) for well-separated data distributions, both NC1 and vanishing test loss are achieved simultaneously. This aligns with the empirical observation that NC1 emerges during training while models attain near-zero test error. Overall, our results demonstrate that NC1 arises from gradient training due to the properties of the loss landscape, and they show the co-occurrence of NC1 and small test error for certain data distributions. 1 Introduction Neural Collapse (NC), first identified by Papyan et al., (2020) , describes a phenomenon observed during the final stages of training where: (i) the penultimate-layer features converge to their respective class means ( NC1 ), (ii) these class means form an equiangular tight frame (ETF) or an orthogonal frame ( NC2 ), and (iii) the columns of the final layer’s classifier matrix similarly form an ETF or orthogonal frame, implementing a nearest class-mean decision rule on the penultimate-layer features ( NC3 ). A popular line of theoretical research has investigated the occurrence of NC via the unconstrained features model (UFM), see ( Fang et al., 2021a, ; Han et al.,, 2022 ; Mixon et al.,, 2022 ) and the discussion in Section 2 . In this framework, the penultimate-layer features are treated as free optimization variables, leading to a benign loss landscape for the resulting optimization problem. The primary justification for adopting the UFM is that the complex feature-learning layers encountered in practice are approximated by a universal learner. While the UFM provides an intriguing theoretical perspective on NC, it has notable limitations. In particular, it neglects the dependence on the data distribution, rendering it unsuitable for theoretically analyzing the relationship between NC during training and the test error ( Hui et al.,, 2022 ) . Furthermore, the training dynamics under the UFM framework is not equivalent to the actual training dynamics of neural networks, which makes it challenging to investigate the occurrence of NC from a dynamical perspective. To address the limitations of UFM, we consider training a three-layer network via gradient flow on the standard mean squared error (MSE) loss. Specifically, we employ a two-layer neural network in the mean-field regime ( Mei et al.,, 2018 ) as the feature-learning component, and then concatenate it with a linear layer as the final predictor. Our main results both (i) establish sufficient conditions on the loss landscape for the first – and most basic – property of neural collapse, i.e., NC1, to hold, and (ii) show that such conditions are in fact satisfied by training the architecture above. This differentiates our paper from recent studies aiming to theoretically explain the NC phenomenon beyond unconstrained features, as existing work either provides only sufficient conditions for NC to occur ( Seleznova et al.,, 2024 ) , focuses on the NTK regime ( Jacot et al.,, 2024 ) , relies on specific training algorithms ( Beaglehole et al.,, 2024 ) or on a specific regularization ( Hong and Ling, 2024a, ) , see Section 2 for a discussion of related work. Specifically, our contributions are summarized below: • First, we connect the emergence of NC1, i.e., the fact that the within-class variability vanishes, with properties of the loss landscape: we show that all approximately stationary points with small empirical loss are roughly NC1 solutions, and the degree to which the within-class variability vanishes is controlled by gradient norm and loss. This implies the prevalence of NC1 during training, as practical training procedures typically converge to such points with small gradient and loss. • Next, we prove that gradient flow on a three-layer network operating in the mean-field regime satisfies the two conditions above (small gradient norm and small loss) and, therefore, it converges to an NC1 solution. While achieving approximately stationary points is expected, the primary challenge lies in controlling the empirical loss due to the model’s non-convex nature. • Finally, we show that, for certain well-separated data distributions, it is possible to achieve NC1 during training as well as vanishing test error, which corroborates the empirical finding that NC1 and strong generalization occur simultaneously. 2 Related work Neural collapse: UFM and beyond. The introduction of the UFM in ( Mixon et al.,, 2022 ; Fang et al., 2021a, ) has prompted a line of work studying the emergence of neural collapse for that model. Specifically, Zhou et al., (2022) focus on the two-layer UFM model, showing that all its stationary points satisfy neural collapse. Han et al., (2022) prove convergence of gradient flow on UFM to NC solutions. Tirer and Bruna, (2022) demonstrate that the global minimizers also satisfy neural collapse when the UFM has multiple linear layers or it incorporates the ReLU activation. Súkeník et al., (2023) extend the results to a deep UFM model for binary classification. Súkeník et al., (2024) then show that, for the deep UFM and multi-class classification, all the global optima still satisfy NC1, but not NC2 and NC3, due to the low-rank bias of the model. We also refer to ( Kothapalli,, 2023 ) for a rather recent and detailed review. Going beyond the UFM, Seleznova et al., (2024) study the connection between NC and the neural tangent kernel (NTK), showing NC under certain block structure assumptions on the NTK matrix. However, the occurrence of such a block structure during training is unclear. Beaglehole et al., (2024) establish NC both empirically and theoretically for Deep Recursive Feature Machine training – a method that constructs a neural network by iteratively mapping the data through the average gradient outer product and then applying an untrained random feature map. Pan and Cao, (2023) consider classification with cross-entropy loss, providing a quantitative bound for NC. Kothapalli and Tirer, (2024) focus on two-layer neural networks in both the NNGP and the NTK limit, proving neural collapse for 1 -dimensional Gaussian data. Hong and Ling, 2024a () study NC for shallow and deep neural networks, also characterizing the generalization error. However, they regularize the loss by the L_{2} -norm of the features rather than the weights, which is different from the weight decay used in practice. Jacot et al., (2024) establish the occurrence of NC for deep neural networks with multiple linear layers, given a balancedness assumption on all the linear layers; in addition, they

Read the complete text