Clustering Properties of Self-Supervised Learning
Xi Weng, Jianing An, Xudong Ma, Binhang Qi, Jie Luo, Xi Yang, Jin Song Dong, Lei Huang 🖂 {}^{~\textrm{\Letter}} · 2025
Read Clustering Properties of Self-Supervised Learning by Xi Weng, Jianing An, Xudong Ma, Binhang Qi, Jie Luo, Xi Yang, Jin Song Dong, Lei Huang 🖂 {}^{~\textrm{\Letter}} (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18452.txt.
Introduction
Xi Weng and co-authors study why self-supervised learning builds such well-clustered internal representations, and how to exploit that strength. Their paper appeared in 2025. These methods learn from unlabeled data by pulling related views together. The team measured clustering at several points inside the model and found the encoder's output clusters best and most steadily.
They built a method, called ReSA, that feeds this clustering strength back into training. The model assigns its own outputs to clusters and learns from those assignments, guiding itself. On standard benchmarks, models trained this way beat leading rivals by a clear margin. Analysis showed clustering improved at both fine and coarse levels. The authors argue that self-supervised models already know how to group; ReSA simply lets them teach themselves.
Catalog record
| CREATOR | Xi Weng, Jianing An, Xudong Ma, Binhang Qi, Jie Luo, Xi Yang, Jin Song Dong, Lei Huang 🖂 {}^{~\textrm{\Letter}} |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18452
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Clustering Properties of Self-Supervised Learning Xi Weng, Jianing An, Xudong Ma, Binhang Qi, Jie Luo, Xi Yang, Jin Song Dong, Lei Huang 🖂 {}^{~\textrm{\Letter}} Abstract Self-supervised learning (SSL) methods via joint embedding architectures have proven remarkably effective at capturing semantically rich representations with strong clustering properties, magically in the absence of label supervision. Despite this, few of them have explored leveraging these untapped properties to improve themselves. In this paper, we provide an evidence through various metrics that the encoder’s output encoding exhibits superior and more stable clustering properties compared to other components. Building on this insight, we propose a novel positive-feedback SSL method, termed Re presentation S elf- A ssignment (ReSA), which leverages the model’s clustering properties to promote learning in a self-guided manner. Extensive experiments on standard SSL benchmarks reveal that models pretrained with ReSA outperform other state-of-the-art SSL methods by a significant margin. Finally, we analyze how ReSA facilitates better clustering properties, demonstrating that it effectively enhances clustering performance at both fine-grained and coarse-grained levels, shaping representations that are inherently more structured and semantically meaningful. 1 Introduction Figure 1: The positive-feedback SSL framework. It involves the model generating representations that possess semantically clustering information. This clustering information is leveraged to design self-supervised loss function, which is then employed to more effectively guide the model’s learning process. Self-supervised learning (SSL) has emerged as a transformative paradigm in universal representation learning ( Oord et al., 2018 ; Bachman et al., 2019 ; He et al., 2020 ; Chen et al., 2020a ; Bao et al., 2021 ; Oquab et al., 2023 ; Assran et al., 2023 ) , consistently surpassing supervised learning in downstream performance. Joint embedding architectures (JEA), in particular, aim to learn invariance of the same data under different transformations and noise ( Bachman et al., 2019 ; He et al., 2020 ; Chen et al., 2020a ) , with demonstrated exceptional effectiveness in visual representation learning. Although such a pretext task may intuitively seem unrelated to capturing semantic relationships, extensive studies ( Caron et al., 2018 ; Caron et al., 2021 ; Assran et al., 2022b ) have demonstrated the strong correlation between its learned representations and semantic information. Ben-Shaul et al. (2023) take a further step to characterize the semantic structures learned by JEA into hierarchic clustering properties, indicating SSL-trained representations exhibit a centroid-like geometric structure and induce three levels of semantic clustering: augmentation sample level, semantic classes, and superclass level. This intriguing finding reveals that SSL methods based on JEA can facilitate strong clustering capabilities during training, but also raises the question of whether these properties hold untapped potential that can be further leveraged to improve SSL itself . Contributions. In this paper, we aim to investigate the design of SSL methods by leveraging the inherent clustering properties of representations, enabling a closed-loop positive-feedback SSL framework, as illustrated in Figure 1 . To achieve this goal, we propose three key questions and our main contributions are summarized as follows: • Where to extract clustering properties from? We demonstrate through various metrics that the encoder’s output, referred to as encoding , exhibits superior and more stable clustering properties compared to other components, such as embedding and the hidden layer outputs within the projector. • How to leverage clustering properties? We propose a novel SSL method, termed Re presentation S elf- A ssignment (ReSA), which employs an online self-clustering mechanism to leverage the model’s inherent clustering properties, thereby facilitating positive-feedback learning. Standard experiments demonstrate that ReSA surpasses existing state-of-the-art methods in both performance and training efficiency. • Whether it facilitates better clustering properties? We examine whether and how ReSA facilitates better clustering properties, demonstrating that it excels at both fine-grained and coarse-grained learning, shaping representations that are inherently more structured and semantically meaningful. 2 Background, Related Work, \& Notation 2.1 Self-Supervised Learning Figure 2: The basic notations for joint embedding architectures (JEA) in SSL. Joint embedding architectures (JEA). Let \mathbf{B} denote a mini-batch input sampled uniformly from a set of images \mathbb{D} , and \mathbb{T} denote the set of data transformations available for augmentation. We consider a pair of neural networks F_{\theta} and F^{\prime}_{\theta^{\prime}} , parameterized by \theta and \theta^{\prime} respectively. They take as input two randomly augmented views, \mathbf{X}=\mathcal{T}(\mathbf{B}) and \mathbf{X}^{\prime}=\mathcal{T}^{\prime}(\mathbf{B}) , where \mathcal{T},\mathcal{T}^{\prime}\in\mathbb{T} ; and they output the embeddings \mathbf{Z}=F_{\theta}(\mathbf{X}) and \mathbf{Z}^{\prime}=F^{\prime}_{\theta^{\prime}}(\mathbf{X}^{\prime}) . The networks are trained with an objective function that minimizes the distances between embeddings obtained from different (two) views of the same images: \displaystyle\mathcal{L}(\mathbf{B},\theta)=\mathbb{E}_{\mathbf{B}\sim\mathbb{D},~\mathcal{T},\mathcal{T}^{\prime}\sim\mathbb{T}}~~\ell\big(F_{\theta}(\mathcal{T}(\mathbf{B})),F^{\prime}_{\theta^{\prime}}(\mathcal{T}^{\prime}(\mathbf{B}))\big). (1) where \ell(\cdot,\cdot) is a loss function, which aims to learn invariance of data transformations. In particular, the JEA usually consist of a shared encoder E_{\theta_{e}}(\cdot) and projector G_{\theta_{g}}(\cdot) , commonly referred to as a Siamese Network ( Chen & He, 2021 ) . As shown in Figure 2 , their outputs \mathbf{H}=E_{\theta_{e}}(\mathbf{X})\in\mathbb{R}^{d_{e}\times m} and \mathbf{Z}=G_{\theta_{g}}(\mathbf{H})\in\mathbb{R}^{d_{g}\times m} are referred to as encoding and embedding , respectively (where m is the mini-batch size, d_{e} and d_{g} are their corresponding feature dimensions). Under this notation, we have F_{\theta}(\cdot)=G_{\theta_{g}}(E_{\theta_{e}}(\cdot)) with learnable parameters \theta=\{\theta_{e},\theta_{g}\} . It is worth noting that extensive work ( Chen et al., 2020a ; Gupta et al., 2022 ) has demonstrated that using the encoding as the representation for downstream tasks achieves much better performance than using embedding , so that the projector is only used during the pre-training process and discarded in inference. Figure 3: Comparison of clustering metrics in encoding \mathbf{H} and embedding \mathbf{Z} across various self-supervised pretrained models. All methods utilize a ResNet-18 encoder pretrained on CIFAR-10 for 1000 epochs. Circular markers represent metrics computed using encodings , while cross markers correspond to metrics derived from embeddings . All metrics are computed on the entire training set, and similar trends can be observed in the validation set. Figure 4: Comparison of linear evaluation accuracy and clustering metrics of encoding \mathbf{H} , embedding \mathbf{Z} , and the hidden layer outputs within the projector \mathbf{P} during the training process. The experiments are conducted using SimCLR, VICReg, and SwAV, employing a ResNet-18 encoder pretrained on CIFAR-100 for 500 epochs. The projector is a standard three-layer MLP with BN and ReLU activations, containing two hidden linear layers, so their outputs are denoted as \mathbf{P}_{0} and \mathbf{P}_{1} . Self-supervised paradigms. The main challenge with JEA is representation collapse, where both branches produce identical and constant outputs regardless of the inputs. Numerous paradigms have been proposed to avoid collapse, including contrastive learning methods ( Chen et al., 2020a ; He et al., 2020 ; Chen et al., 2020b ; Chen et al., 2021 ; Liu et al., 2022 ) that attract different views from the same image (positive pairs) while pushing apart different images (negative pairs), and non-contrastive approaches ( Grill et al., 2020 ; Caron et al., 2020 ; Caron et al., 2021 ; Weng et al., 2024 ) which directly align positive targets without incorporating negative pairs. Although Ben-Shaul et al. (2023) had demonstrated that the encodings learned through SSL are highly correlated with semantic classes and exhibit strong clustering capabilities, few methods have leveraged this clustering ability to facilitate positive-feedback learning. A closely related work ( Ma et al., 2023 ) exploits the encoding ’s augmentation robustness to re-weight the positive alignment in the SSL objective functions;