Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research
Patrik Reizinger, Randall Balestriero, David Klindt, Wieland Brendel · 2025
Read Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research by Patrik Reizinger, Randall Balestriero, David Klindt, Wieland Brendel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13101.txt.
Introduction
Self-supervised learning powers much of modern AI, and its methods seem to land on the same learned forms, a pattern the Platonic representation hypothesis describes. The four authors argue in this position paper that identifiability theory can explain that likeness, but only after the theory grows. Its results so far do not explain why these methods work so well in practice.
They propose Singular Identifiability Theory, a frame that covers the whole chain of steps. It would bring out the hidden ideas about the data that these methods rest on. It would push toward learned forms that are easier to read and hold up more widely. Three paths come first: how training runs and settles, and the role of finite samples, batch size, and how much the data vary. The third is the leanings built into model shapes, ways the data are changed, starting points, and tuning tools.
Catalog record
| CREATOR | Patrik Reizinger, Randall Balestriero, David Klindt, Wieland Brendel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13101
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research Patrik Reizinger, Randall Balestriero, David Klindt, Wieland Brendel Abstract Self-Supervised Learning (SSL) powers many current AI systems. As research interest and investment grow, the SSL design space continues to expand. The Platonic view of SSL, following the Platonic Representation Hypothesis (PRH), suggests that despite different methods and engineering approaches, all representations converge to the same Platonic ideal. However, this phenomenon lacks precise theoretical explanation. By synthesizing evidence from Identifiability Theory (IT), we show that the PRH can emerge in SSL. There is a gap between SSL theory and practice: Current IT cannot explain SSL’s empirical success, though it has practically relevant insights. Our work formulates a blueprint for SSL research to bridge this gap: we propose expanding IT into what we term Singular Identifiability Theory (SITh), a broader theoretical framework encompassing the entire SSL pipeline. SITh would allow deeper insights into the implicit data assumptions in SSL and advance the field towards learning more interpretable and generalizable representations. We highlight three critical directions for future research: 1) training dynamics and convergence properties of SSL; 2) the impact of finite samples, batch size, and data diversity; and 3) the role of inductive biases in architecture, augmentations, initialization schemes, and optimizers. 1 Introduction Self-supervised learning (SSL) drives many current AI breakthroughs in language, vision, and image-text models ( Radford et al., 2018 ; Radford et al., 2021 ; Assran et al., 2023 ) . This remarkable success builds on a plethora of algorithms and engineering practices. However, the model zoo makes SSL hard to navigate, divides the attention for a better theoretical understanding and, we argue, dampens future advancements. SSL is likely to continue being a foundational paradigm of machine learning, but it needs to be consolidated, similar to the program of unifying deep learning through geometry ( Bronstein et al., 2021 ) . There were recent attempts to unify the SSL model zoo ( Wang & Isola, 2020 ; Morningstar et al., 2024 ; Bizeul et al., 2024 ; Fleissner et al., 2025 ) . Huh et al. (2024) even argued, by positing the Platonic Representation Hypothesis (PRH) , that neural networks converge to effectively the same representation, despite different setups, loss functions, data modalities, and engineering practices. However, their analysis does not provide a conclusive mathematical answer when and why this happens. We do have theories that can provide such an answer, but a gap exists between SSL theory and practice. Prevalent frameworks such as identifiability theory (IT) cannot explain the role of initialization, stop gradients, or learning dynamics, and neither can solve problems like dimensional collapse ( Jing et al., 2022 ) or loss saturation. An even larger problem of current theory is generalization: The promise of self-supervised representations to work well on many downstream tasks alludes to “universality,” or at least out-of-distribution (OOD) generalization. However, current theories mostly focus on the i.i.d. case, and empirical methods generally perform well only in-distribution ( Montero et al., 2021 ; Schott et al., 2021 ; Montero et al., 2024 ; Mayilvahanan et al., 2024a ; Mayilvahanan et al., 2024b ) . Position: empirical advancements alone are insufficient to accelerate SSL research, we need to bridge the gap between theory and practice. We need an empirically grounded theory, which we call Singular Identifiability Theory (SITh) , to accelerate SSL research by: (i) Designing realistic data generating processes (DGPs) , grounded in empirical observations; (ii) Formalizing when and why self-supervised representations are converging; (iii) Providing principled recommendations for designing and evaluating SSL . This position paper argues that to accelerate SSL research, we need to develop a theoretical framework grounded in empirical observation. Acknowledging the success of IT , we call for its extension to what we term Singular Identifiability Theory (SITh) —alluding to Singular Learning Theory (SLT) ( Watanabe, 2009 ; Watanabe, 2020 ) , which extended learning theory (LT) to considerations relevant to neural networks. SSL practitioners need to rely on principled guidance derived from theory, whereas theoreticians need to concentrate on studying practically relevant aspects of SSL —we provide concrete examples and research questions in Tab. 1 . Our contributions are: • We provide an approachable introduction to identifiability theory (IT) for practitioners, demonstrating how SSL practice can benefit from IT ( § 3 ); • We call for an empirically grounded extension of identifiability, termed Singular Identifiability Theory (SITh) to close the gap between SSL theory and practice, mainly relying on the concept of a DGP ( § 4 ); • We provide an extensive overview of the open questions in SSL, formulate concrete research questions and promising research directions to suggest how empirical observations and theoretical results can drive SSL forward ( Tab. 1 ). 2 Background SSL drives most AI breakthroughs. Long gone are the days when supervised, semi-supervised, and unsupervised learning co-existed as the main tracks of top AI conferences. The emergence of Internet-scale datasets coupled with the need for foundation models ( Bommasani et al., 2021 ) —models capable of solving many tasks post-training—has moved the landscape of AI under one umbrella: self-supervised learning (SSL) . With access to data and compute, hundreds of SSL methods have emerged ( Balestriero et al., 2023 ) . As a result, SSL now holds the state-of-the-art across numerous domains like remote sensing and medical data ( Wang et al., 2022 ; Krishnan et al., 2022 ) , and applications such as zero-shot reasoning ( Radford et al., 2021 ; Zhai et al., 2023 ) , multi-modal learning ( Girdhar et al., 2022 ) . The distinct engineering choices and implementation details make it hard to empirically test every hypothesis one may have in mind. Yet, such hypothesis testing is becoming more important than ever to understand possible robustness, fairness, compositional, or structural limitations of existing methods. Without a clear understanding of today’s limitations, it is getting harder to develop tomorrow’s methods. Self-supervised representations are often but not always similar. Theoretical and empirical evidence indicates a surprising similarity between some self-supervised representations. Roeder et al. (2020) proved that in specific cases, two trained models learn a similar representation up to a linear representation. Works on relative representations and model stitching ( Moschella et al., 2023 ; Cannistraci et al., 2023 ; Norelli et al., 2023 ; Fumero et al., 2024 ; Maiorca et al., 2024 ) demonstrated that learned self-supervised representations can be related via simple (e.g., linear) transformations. A similar notion of representation linearity was demonstrated in language models ( Park et al., 2023 ; Marconato et al., 2024 ; Jha et al., 2025 ) , building from earlier observations of analogy making in word embeddings like word2vec ( Mikolov et al., 2013 ; Arora et al., 2016 ) . The PRH ( Huh et al., 2024 ) proposed an explanation of why such convergence can happen, alluding to Plato’s allegory of the cave. Huh et al. (2024) formalized three hypotheses, corresponding to the effect of: i) function class , ii) data sets and loss functions , and iii) regularization . Earlier work by Duan et al. (2020) suggested a link between model convergence and identifiability. We discuss evidence for their importance in SSL and highlight that