Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking

Ting Han, Linara Adilova, Henning Petzka, Jens Kleesiek, Michael Kamp · 2025

Read Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking by Ting Han, Linara Adilova, Henning Petzka, Jens Kleesiek, Michael Kamp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17738.txt.

Introduction

Ting Han, Linara Adilova, Henning Petzka, Jens Kleesiek, and Michael Kamp pull apart two darlings of deep learning theory. Their 2025 arXiv paper concerns neural collapse, where class representations settle into a neat symmetric pattern, and flatness of the loss landscape. Both often appear near good generalization, and both have been cast as its cause.

The team uses grokking, the regime where a model memorizes first and generalizes much later, to separate cause from coincidence in time. Both collapse and flatness show up as generalization begins. Only flatness, though, predicts it across their tests. Models pushed toward or away from collapse generalize alike, while models steered away from flat solutions stall, even where grokking never normally occurs. A theory result ties collapse to flatness under classic assumptions. The verdict: flatness is necessary, collapse is not.

Catalog record

CREATORTing Han, Linara Adilova, Henning Petzka, Jens Kleesiek, Michael Kamp
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-17738

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking Ting Han, Linara Adilova, Henning Petzka, Jens Kleesiek, Michael Kamp Abstract Neural collapse, i.e., the emergence of highly symmetric, class-wise clustered representations, is frequently observed in deep networks and is often assumed to reflect or enable generalization. In parallel, flatness of the loss landscape has been theoretically and empirically linked to generalization. Yet, the causal role of either phenomenon remains unclear: Are they prerequisites for generalization, or merely by-products of training dynamics? We disentangle these questions using grokking, a training regime in which memorization precedes generalization, allowing us to temporally separate generalization from training dynamics and we find that while both neural collapse and relative flatness emerge near the onset of generalization, only flatness consistently predicts it. Models encouraged to collapse or prevented from collapsing generalize equally well, whereas models regularized away from flat solutions exhibit delayed generalization. Furthermore, we show theoretically that neural collapse implies relative flatness under classical assumptions, explaining their empirical co-occurrence. Our results support the view that relative flatness is a potentially necessary and more fundamental property for generalization, and demonstrate how grokking can serve as a powerful probe for isolating its geometric underpinnings. 1 Introduction Overparameterized neural networks continue to challenge classical learning theory. Despite their capacity to memorize arbitrary labels ( Zhang et al. 2016 ) , they often generalize well on natural data and, in some cases, only begin to generalize after complete memorization, a phenomenon known as grokking ( Power et al. 2022 ) . This apparent contradiction has motivated a renewed interest in identifying geometric signatures of generalization. Two candidates have emerged as particularly prominent: the flatness of the loss landscape ( Keskar et al. 2017 ; Jiang et al. 2020 ; Petzka et al. 2021 ) and the neural collapse (NC) phenomenon ( Papyan et al. 2020 ; Mixon et al. 2022 ; Zhou et al. 2022 ) . Both tend to appear late in training and are frequently associated with good generalization, yet their precise causal roles remain unclear. In this paper, we challenge the common conjecture that NC is necessary for generalization ( Mixon et al. 2022 ; Zhou et al. 2022 ; Súkeník et al. 2023 ; Zhu et al. 2021 ) . Our key idea is to leverage grokking as a unique observational window: since generalization emerges only after prolonged memorization, it enables a clean separation between training dynamics and generalization. This allows us to ask—do neural collapse and flatness merely correlate with generalization, or do they contribute to it? We measure both phenomena in the penultimate layer. For flatness, we adopt the relative flatness metric proposed by Petzka et al. 2021 , which considers the Hessian trace normalized by the weight norm, a quantity theoretically and empirically shown to align with generalization. To make this computable in large state-of-the-art neural networks, we employ the alternative closed-form upper bound introduced by ( Walter et al. 2024 ) , which is valid in the penultimate layer under cross-entropy loss. For NC, we measure the class-wise clustering of penultimate-layer representations through empirical variance and mean properties, tracking its emergence throughout training. In grokking experiments, we find that flatness emerges alongside generalization, never before. NC, in contrast, appears earlier during memorization, suggesting that it may result from, rather than enable, generalization (cf. Fig. 1 ). Figure 1: Neural collapse clustering and relative flatness in grokking. While both correlate with generalization, neural collapse emerges early during memorization, whereas flatness only drops sharply when generalization begins, highlighting flatness as a better indicator of generalization onset. To isolate their functional roles, we perturb training dynamics. When we suppress collapse-style clustering, networks still generalize robustly; test accuracy remains high despite high clustering values. Flatness, however, is unaffected. In contrast, regularizing against flatness, i.e., encouraging sharp solutions, reliably delays generalization. These findings point to flatness as a more fundamental driver of generalization. To explain the frequent co-occurrence of NC and flatness, we provide a theoretical result: under classical NC assumptions, collapse implies relative flatness. This unifies previously observed correlations under a single geometric framework, but also shows that flatness can arise via other mechanisms. Neural collapse is thus a sufficient but non-exclusive path toward flatness. Finally, we revisit the theoretical framework of Petzka et al. 2021 , which asserts that generalization requires not only flatness, but also representativeness , i.e., the alignment between learned features and the true data distribution. Our experiments confirm this dependency. Representativeness, when approximated, improves only as generalization sets in, highlighting that neither flatness nor collapse alone ensures generalization without meaningful feature alignment. Together, these results place neural collapse in a new light. While it often accompanies generalization, it is not a prerequisite. Instead, it may serve as an inductive bias toward flatness under standard training dynamics. Relative flatness, by contrast, appears empirically necessary and theoretically justified, especially when labels are locally constant and features are representative—assumptions which appear to hold in our experimental settings. Our work provides new insight into the geometry of generalization: • We show empirically that relative flatness, but not neural collapse, is potentially necessary for generalization; • We prove that neural collapse implies relative flatness under classical assumptions; • We demonstrate that delayed generalization can be actively induced by manipulating flatness, even in real-world architectures. These insights challenge prevailing narratives about generalization in deep learning and suggest new levers for training, regularization, and model diagnostics. 2 Related Work Despite substantial empirical success, the mechanisms driving generalization in deep neural networks remain poorly understood. Two pronounced geometric perspectives, neural collapse and loss surface flatness , have emerged as phenomena empirically correlated with generalization, but their interrelation and causal roles remain ambiguous. In this work, we leverage the properties of grokking to disentangle necessity from sufficiency and to assess which of these phenomena are fundamentally required for generalization. Neural Collapse and Generalization Neural collapse (NC) describes a geometric convergence during training a classifier where penultimate layer features from the same class collapse to a single mean, the means form a simplex equiangular tight frame, and classifier weights align accordingly ( Papyan et al. 2020 ) . This phenomenon has been widely observed at late training stages in overparameterized networks ( Han et al. 2021 ; Graf et al. 2021 ; Rangamani and Banburski-Fahey 2022 ; Wu and Papyan 2024 ; Zhou et al. 2025 ) and has become a widely accepted indicator for generalization and robustness. Under idealistic assumption of an unconstrained features model, i.e., a model which optimizes features as free variables, NC has been shown to be the global optimal solution for various loss functions ( Mixon et al. 2022 ; Zhou et al. 2022 ; Súkeník et al. 2023 ; Zhu et al. 2021 ) . Similar results for compatibility of zero test error and collapse of variances (so only NC1 condition) were shown for mean-field networks ( Wu and Mondelli 2025 ) . Practical applications have leveraged NC properties for downstream performance. For instance, Wu et al. 2025 use a feature separation loss based on NC to improve out-of-distribution detection. Similarly, Munn et al. 2024 show that lower geometric complexity in pre-trained representations fosters NC and leads to better few-shot generalization in

Read the complete text