Clustering via Self-Supervised Diffusion
Roy Uziel, Irit Chelly, Oren Freifeld, Ari Pakman · 2025
Read Clustering via Self-Supervised Diffusion by Roy Uziel, Irit Chelly, Oren Freifeld, Ari Pakman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04283.txt.
Introduction
Roy Uziel, Irit Chelly, Oren Freifeld, and Ari Pakman turn diffusion models, best known for generation work, to the grouping task. Their 2025 frame, CLUDI, pairs the diffusion craft with traits from a pre-trained vision model. A teacher-student pair drives the learning: the teacher generates wide, chance-spread group calls, and the student firms them into steady calls. The chance in the generation acts as a fresh augmentation kind.
That play lets the frame find fine grain in thick data that stiff methods miss. Tests on hard sets set new top marks for label-free grouping, in accuracy and in toughness both. Diffusion craft, the team holds, has a second life past generation: it can sort the world it learned to fake.
Catalog record
| CREATOR | Roy Uziel, Irit Chelly, Oren Freifeld, Ari Pakman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-04283
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Clustering via Self-Supervised Diffusion Roy Uziel, Irit Chelly, Oren Freifeld, Ari Pakman Abstract Diffusion models, widely recognized for their success in generative tasks, have not yet been applied to clustering. We introduce Clustering via Diffusion (CLUDI), a self-supervised framework that combines the generative power of diffusion models with pre-trained Vision Transformer features to achieve robust and accurate clustering. CLUDI is trained via a teacher-student paradigm: the teacher uses stochastic diffusion-based sampling to produce diverse cluster assignments, which the student refines into stable predictions. This stochasticity acts as a novel data augmentation strategy, enabling CLUDI to uncover intricate structures in high-dimensional data. Extensive evaluations on challenging datasets demonstrate that CLUDI achieves state-of-the-art performance in unsupervised classification, setting new benchmarks in clustering robustness and adaptability to complex data distributions. Our code is available at https://github.com/BGU-CS-VIL/CLUDI . 1 Introduction Clustering is a fundamental task in unsupervised learning, essential for uncovering meaningful groupings within data. These groupings play a vital role in diverse downstream applications, such as image segmentation ( Mittal et al. 2022 ; Friebel et al. 2022 ) , anomaly detection ( Song et al. 2021a ) , and bioinformatics ( Karim et al. 2021 ) . Despite significant advancements, traditional methods face significant challenges, particularly in datasets with intricate structures and varying intra-class similarity, where such approaches often struggle to capture underlying patterns ( Ben-David 2018 ) . Figure 1 : Overview of the CLUDI Framework at Inference. Images pass through a pre-trained Vision Transformer to obtain their feature representations. Multiple random vectors are sampled from a Gaussian distribution. A diffusion model, conditioned on the features, refines the random vectors into class assignment embeddings. Each refined embedding corresponds to a candidate cluster probability vector. By averaging multiple such assignments, the framework produces robust and accurate clustering predictions. To address these limitations, deep learning-based clustering approaches have gained substantial attention for their ability to tackle complex and diverse data landscapes ( Zhou et al. 2024 ; Ren et al. 2024 ; Wei et al. 2024 ) . However, deep learning-based clustering methods still face persistent challenges that limit their practical utility. A primary issue is model collapse ( Amrani et al. 2022 ) , where learned representations degenerate into trivial solutions. Another obstacle arises from underutilizing pre-trained features, resulting in suboptimal performance. Notably, Adaloglou et al. 2023 demonstrate that using pre-trained Vision Transformers ( Caron et al. 2021 ) can surpass methods that attempt to learn both representations and cluster assignments simultaneously, thereby highlighting the effectiveness of high-quality feature initialization. Self-supervised representations have consistently outperformed supervised ones in transfer tasks ( Ericsson et al. 2021 ) , highlighting the benefits of leveraging powerful pre-trained models. Meanwhile, many clustering frameworks rely on complex augmentation pipelines, such as neighbor mining ( Van Gansbeke et al. 2020 ; Adaloglou et al. 2023 ) , or employ multiple clustering heads trained in parallel, selecting the best-performing one at evaluation time ( Adaloglou et al. 2023 ; Amrani et al. 2022 ) . Although these strategies can improve accuracy, they introduce additional complexity and computational overhead during training. To address these limitations, we introduce Clustering via Diffusion (CLUDI) , a self-supervised framework that leverages the generative strengths of diffusion models to produce robust cluster probabilities. As Figure 1 illustrates, CLUDI takes as input a pre-trained Vision Transformer feature vector \mathbf{x}\in\mathbb{R}^{n} and refines an initial random assignment embedding vector through an iterative diffusion process. Conditioned on \mathbf{x} , this process evolves over multiple steps and culminates in a final assignment embedding \mathbf{z}_{0}\in\mathbb{R}^{d} , which encodes the likelihood of each data point belonging to each of the K clusters. During inference, CLUDI generates multiple stochastic samples of \mathbf{z}_{0} and aggregates the corresponding predictions. By averaging across diverse representations, this strategy mitigates uncertainty, uncovers subtle structures in high-dimensional feature spaces, and yields more stable and accurate cluster assignments, even in complex and diverse data scenarios. Our model is trained using a self-supervised Siamese architecture comprising teacher and student branches. The teacher, implemented as a diffusion model, generates assignment embeddings \mathbf{z}_{0}\in\mathbb{R}^{d} and cluster probabilities p(k|\mathbf{x}) for k\in\{1,2,\ldots,K\} , through stochastic sampling. These outputs, which capture diverse and complementary views of the data, serve as targets for the student. The student adapts its predictions to match the teacher’s outputs, enabling it to uncover meaningful and distinct clusters in the data. The training process optimizes two complementary objectives: an asymmetric non-contrastive loss ( Grill et al. 2020 ; Chen & He 2021 ; Caron et al. 2021 ) for the assignment embedding \mathbf{z}_{0} and a non-collapsing cross-entropy loss for the cluster probabilities p(k|\mathbf{x}) . To facilitate effective clustering and prevent trivial solutions, the cross-entropy loss incorporates a uniform prior over the data mini-batch ( Amrani et al. 2022 ) , encouraging well-separated and diverse cluster assignments. Why use diffusion for clustering? Despite their success in generative tasks, diffusion models have not been explored for clustering until now. We argue, however, that their ability to model and sample from complex, high-dimensional distributions makes them particularly suited for clustering high-dimensional data, such as images. By iteratively refining noisy representations, diffusion models can uncover underlying structure and variability in the data, providing a natural mechanism for capturing meaningful cluster assignments. Their stochastic nature further enables robust and diverse clustering predictions. Our key contributions are as follows: • We introduce Clustering via Diffusion (CLUDI) , a novel framework that is the first to leverage diffusion models for the clustering task. • We propose a self-supervised training paradigm based on a teacher-student architecture, where a diffusion model generates stochastic and informative cluster assignments to guide the student in learning meaningful cluster probabilities. • We demonstrate the effectiveness of CLUDI through extensive evaluations on benchmark datasets, achieving state-of-the-art accuracy and robustness in unsupervised classification tasks. 2 Related Work Deep learning-based models that learn to cluster are usually referred to as performing deep clustering or unsupervised classification . Cluster categories are learned during the training phase and remain fixed during inference. These methods have received much interest in the machine learning community. Comprehensive reviews on this vast field can be found in ( Zhou et al. 2024 ; Ren et al. 2024 ; Wei et al. 2024 ) . End-to-end training. Most previous works learn simultaneously feature representations and cluster categories. An early work in this area is Deep Embedded Clustering (DEC) ( Xie et al. 2016 ) , which jointly optimizes feature learning through an autoencoder and assigns clusters using a Kullback-Leibler (KL) divergence-based loss. Although DEC can be effective in learning clusters, its performance is sensitive to initialization and prone to model collapse. DeepCluster ( Caron et al. 2018 ) takes a different approach by alternating between pseudo-label generation and feature representation refinement. The method iteratively updates cluster centroids and uses the centroids to assign pseudo-labels, which are then used to optimize the feature space. However, the computational overhead associated with generating and updating pseudo-labels can be significant, especially for large-scale datasets. Invariant Information Clustering (IIC) ( Ji et al. 2019 ) offers a complementary method by maximizing mutual information between different augmentations of the same data. Among probabilistic approaches, Variational Deep Embedding (VaDE) ( Jiang et al. 2016 ) integrates variational autoencoders with Gaussian mixture models to