Keep it Light! Simplifying Image Clustering via Text-Free Adapters

Yicen Li, Haitz Sáez de Ocáriz Borde, Anastasis Kratsios, Paul D. McNicholas · 2025

Read Keep it Light! Simplifying Image Clustering via Text-Free Adapters by Yicen Li, Haitz Sáez de Ocáriz Borde, Anastasis Kratsios, Paul D. McNicholas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04226.txt.

Introduction

Grouping images into clusters without labels usually takes a heavy pipeline, often with a text model in the loop and text-image pairs that real users may not have. Li and co-authors show that a far lighter recipe can match those systems. Their method, called SCP, keeps a pre-trained vision model frozen and trains only a small cluster head on its features, using pairs of related images as the teaching signal.

Across six standard benchmarks, from CIFAR sets to ImageNet subsets, the simple method scored close to the best complex pipelines. The authors also prove a theory result explaining why, at least under ideal conditions, text embeddings add nothing essential for clustering images. They pitch the work as a cheaper, easier path to image clustering that labs without large compute can actually run.

Catalog record

CREATORYicen Li, Haitz Sáez de Ocáriz Borde, Anastasis Kratsios, Paul D. McNicholas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-04226

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Keep it Light! Simplifying Image Clustering via Text-Free Adapters Yicen Li, Haitz Sáez de Ocáriz Borde, Anastasis Kratsios, Paul D. McNicholas Abstract In the era of pre-trained models, effective classification can often be achieved using simple linear probing or lightweight readout layers. In contrast, many competitive clustering pipelines have a multi-modal design, leveraging large language models (LLMs) or other text encoders, and text-image pairs, which are often unavailable in real-world downstream applications. Additionally, such frameworks are generally complicated to train and require substantial computational resources, making widespread adoption challenging. In this work, we show that in deep clustering, competitive performance with more complex state-of-the-art methods can be achieved using a text-free and highly simplified training pipeline. In particular, our approach, Simple Clustering via Pre-trained models (SCP), trains only a small cluster head while leveraging pre-trained vision model feature representations and positive data pairs. Experiments on benchmark datasets, including CIFAR-10, CIFAR-20, CIFAR-100, STL-10, ImageNet-10, and ImageNet-Dogs, demonstrate that SCP achieves highly competitive performance. Furthermore, we provide a theoretical result explaining why, at least under ideal conditions, additional text-based embeddings may not be necessary to achieve strong clustering performance in vision. I Introduction Powered by the expressive capabilities of neural networks, deep clustering (DC) has redefined the state of the art (SOTA) in image clustering, outperforming classical algorithms by indisputable margins. Modern DC pipelines uncover subtle feature structures that substantially enhance the effectiveness of downstream learners. A highly non-exhaustive list of domains where DC-powered algorithms have reshaped the SOTA includes community detection [ 1 , 2 ] , anomaly detection [ 3 ] , image segmentation [ 4 , 5 ] , object detection [ 6 , 7 ] , and various medical applications [ 8 ] . However, in addition to image encoders, many recent DC pipelines rely on massive models—often involving large language models (LLMs)—that are unnecessarily compute-intensive for clustering tasks and introduce complicated training schemes. Unlike supervised learning methods, which typically require only a lightweight readout layer or linear probing for classification, current unimodal deep clustering frameworks frequently incorporate additional components such as support sets, feature projection layers, or teacher–student architectures. While these designs have achieved impressive performance, they also introduce extra hyperparameters, increase training complexity, and ultimately raise an important research question. I-A Is SOTA performance achievable with a simple DC pipeline? As image clustering often serves as a foundational step for a variety of downstream tasks, a simple yet effective solution can thus provide broad utility to the community. Indeed, we show that competitive performance, SOTA and near-SOTA, can be achieved using only a small adapter modifying the features generated by a pre-trained text-free, image encoder, i.e. not requiring any additional input text from the user, LLMs, or any other variants of text encoders. The result is a lightweight and simple end-to-end DC pipeline, which is computationally cheap enough to be run on a standard L4 GPU. In parallel to how linear probing or read-out layers are deployed in classification tasks, we show that our adapter is easy to adopt in real-world applications, as it is simple and effective. I-B Theoretical Motivation Text-image pairs may not always be readily available in practical clustering applications. The intuition behind why a text-free classifier should, at least theoretically, be able to match the power of a classifier using text embeddings together with the raw pixel data is rooted in our theoretical results. If X represents a random image and Z is its corresponding semantic description, which itself is a compressed representation of the information only in the random image X , then any classification task Y that depends on both the image X and its textual information can ultimately be expressed as a function of the image X alone. We denote the \sigma -algebra generated by the random variable X by \sigma(X) . Proposition 1 ( Lossless Amortization Principle (LAR) ) . Fix d,D,C\in\mathbb{N} . Let X,Z be random variables respectively, taking values in \mathbb{R}^{d} and in \mathbb{R}^{D} , both of which are defined on a common probability space (\Omega,\mathcal{F},\mathbb{P}) , and suppose that Z is \sigma(X) -measurable. For every \{0,1,\dots,C-1\} -valued random variable Y on (\Omega,\mathcal{F},\mathbb{P}) if: there exists a Borel measurable function f:(\mathbb{R}^{d+D},\mathcal{B}(\mathbb{R}^{d+D}))\to(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d})) representing Y by \underbrace{Y=f(X,Z)}_{\text{Depends both on $X$ and $Z$}} (1) then, there is a Borel map F:(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}))\to(\mathbb{R}^{n},\mathcal{B}(\mathbb{R}^{n})) providing the following lossless amortized representation Y=\underbrace{F(X).}_{\text{Only depends on $X$}} (LAR) Proof of Proposition 1 . By the Doob–Dyknin lemma, see e.g. [ 9 , Lemma 1.14] , since Z is \sigma(X) -measurable iff there exists some Borel measurable function g:(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}))\to(\mathbb{R}^{D},\mathcal{B}(\mathbb{R}^{D})) such that Z=g(X). (2) Define F:(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}))\to(\mathbb{R}^{n},\mathcal{B}(\mathbb{R}^{n})) by composition: F(x)\triangleq f(x,g(x)). (3) Then, by ( 2 ) and ( 1 ), Y=f(X,Z)=f(X,g(X))=F(X), which establishes the claim. ∎ Further, the proposition above seems to align with the Platonic Representation Hypothesis proposed in [ 10 ] , which suggests that latent representations in deep networks converge despite being trained on distinct data modalities. This supports our claim that, as long as the backbone vision model provides sufficiently good latent representations, strong performance can be achieved through a simple DC pipeline without relying on additional modality inputs. We acknowledge that, generally speaking, incorporating multimodal representations and additional information in our network tends to yield better empirical performance. Rather than disputing this, we deliberately refrain from using such frameworks, and instead aim to explore whether, in principle and under ideal conditions, clustering could be achieved using the simplest model possible, without multimodal data collection. Our LAR principle (Proposition 1 ) shows that any semantically-powered image classifier may be realized by a text-free classifier “ F ”. However, can this theoretical classifier be practically realized, at-least approximately? Our main theoretical result guarantees that the theoretical text-free classifiers can indeed be approximately implemented by an MLP with a \operatorname{softmax} output activation. Our guarantee holds for multiclass MLP classifiers using any standard activation function such as \operatorname{ReLU} , \operatorname{Swish} , or \operatorname{softplus} . Theorem 1 ( Text-Free DC is Powerful Enough ) . In the setting of Proposition 1 , let \rho:\mathbb{R}\to\mathbb{R} be an activation function with at least one point of continuous and non-zero differentiability. Then, for every 0<\varepsilon\leq 1 , there is an MLP \hat{F}:\mathbb{R}^{d}\to\mathbb{R}^{C} , with activation function \rho , satisfying \big|\underbrace{f(X,Z)}_{\text{text-dependent classes}}-\underbrace{\,\,\operatorname{softmax}\circ\hat{F}(X)}_{\text{text-free deep learner}}\big|<\varepsilon (4) with probability at least 1-\varepsilon . Proof of Theorem 1 . Let \mu\triangleq X_{\#}\mathbb{P} denote the law of X . By [ 11 , Theorem 13.6] , \mu is a Radon measure. Since \mathbb{R}^{d} is second-countable and locally compact, Lusin’s theorem [ 11 , Exercise 13.1.3] yields, for every \varepsilon\in(0,1] , a compact set \mathcal{K}_{\varepsilon}\subset\mathbb{R}^{d} such that \mu(\mathcal{K}_{\varepsilon})\geq 1-\varepsilon, (5) and F is continuous on \mathcal{K}_{\varepsilon} . Now, let \rho satisfy [ 12 , Assumption 1] (as in [ 13 ] ), and recall that the softmax activation satisfies [ 12 , Assumption 8] . Then, the universal approximation theorem [ 12 , Theorem 37(ii)] ensures the existence of an MLP \hat{F}:\mathbb{R}^{d}\to\mathbb{R}^{C} such that \sup_{x\in\mathcal{K}_{\varepsilon}}\big|F(x)-\operatorname{softmax}\circ\hat{F}(x)\big|<\varepsilon. (6) Define B\triangleq\bigl\{x\in\mathbb{R}^{d}:\,|F(x)-\operatorname{softmax}\circ\hat{F}(x)|\geq\varepsilon\bigr\}. Then \mathcal{K}_{\varepsilon}\subseteq\mathbb{R}^{d}\setminus B . Let L(x)=|F(x)-\operatorname{softmax}\circ\hat{F}(x)| . Since L is measurable, so is B=L^{-1}([\varepsilon,\infty)) . Combining ( 5 ) and ( 6 ), we get \begin{aligned} \mathbb{P}\!\Big(|F(X)-\operatorname{softmax}\!\circ\!\hat{F}(X)|<\varepsilon\Big)&=\mu\!\Big(|F(x)-\operatorname{softmax}\!\circ\!\hat{F}(x)|<\varepsilon\Big)\\ &=\mu(\mathbb{R}^{d}\setminus B)\\ &\geq\mu(\mathcal{K}_{\varepsilon})\\ &\geq 1-\varepsilon.\end{aligned} Finally, by Proposition 1 and ( 1 ), F(X)=f(X,Z). (7) Substituting ( 7 ) into (

Read the complete text