ESMC: MLLM-Based Embedding Selection for Explainable Multiple Clustering
Xinyue Wang, Yuheng Jia, Hui Liu, Junhui Hou · 2025
Read ESMC: MLLM-Based Embedding Selection for Explainable Multiple Clustering by Xinyue Wang, Yuheng Jia, Hui Liu, Junhui Hou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00725.txt.
Introduction
Xinyue Wang, Yuheng Jia, Hui Liu, and Junhui Hou wrote a 2025 paper on clustering, the task of grouping data points that belong together. Most deep clustering methods give just one grouping for a dataset. They assume the data follows one fixed pattern, which may not match what a user needs.
Wang and her team present ESMC, which uses a large multimodal model to select embeddings for explainable multiple clustering. Embeddings are numeric summaries that capture what each item is like. Producing several good groupings, each with an explanation, lets users pick the view that fits their question. That matters in fields where one dataset holds many stories.
Catalog record
| CREATOR | Xinyue Wang, Yuheng Jia, Hui Liu, Junhui Hou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-00725
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ESMC: MLLM-Based Embedding Selection for Explainable Multiple Clustering Xinyue Wang, Yuheng Jia, Hui Liu, Junhui Hou Abstract Typical deep clustering methods, while achieving notable progress, can only provide one clustering result per dataset. This limitation arises from their assumption of a fixed underlying data distribution, which may fail to meet user needs and provide unsatisfactory clustering outcomes. Our work investigates how multi-modal large language models (MLLMs) can be leveraged to achieve user-driven clustering, emphasizing their adaptability to user-specified semantic requirements. However, directly using MLLM output for clustering has risks for producing unstructured and generic image descriptions instead of feature-specific and concrete ones. To address these issues, our method first discovers that MLLMs’ hidden states of text tokens are strongly related to the corresponding features, and leverages these embeddings to perform clusterings from any user-defined criteria. We also employ a lightweight clustering head augmented with pseudo-label learning, significantly enhancing clustering accuracy. Extensive experiments demonstrate its competitive performance on diverse datasets and metrics. Codes and datasets are available in github: Code — https://github.com/JCSTARS/Embedding-Selective-Multiple-Clustering 1 Introduction Clustering is an unsupervised machine learning technique focused on grouping objects based on their specific patterns ( 18 ; 43 ) . Traditional clustering methods ( 28 ; 10 ) often rely on hand-crafted features, which can face difficulties with real-world, high-dimensional data. Deep clustering ( 42 ; 40 ; 4 ) addresses this by leveraging the strong representation learning power of deep neural networks to produce feature embeddings that are more conducive to clustering. Most of these methods group the dataset with a single standard. However, users may require clustering a set of images based on multiple criteria, as illustrated in Figure 1 . For instance, a set of car images can be categorized according to different attributes such as color, brand, or body style. This scenario highlights the challenge of multi-clustering, where diverse grouping objectives coexist for the same dataset. Figure 1: Multiple clustering of sample images on the Stanford_cars dataset with cluster criterion of color, manufacturer, and body style. Multiple clustering demonstrates that a single dataset can be partitioned into distinct clusters based on different underlying structures ( 47 ) . While existing methods have offered solutions for obtaining diverse clusters ( 1 ; 8 ; 39 ; 44 ) , they often rely on inherent data characteristics and lack the flexibility to directly incorporate user-defined criteria. Foundational models like CLIP ( 37 ) recently enable clustering tasks ( 46 ; 45 ) by aligning image and text features. However, CLIP’s prompt-agnostic image embeddings limit clustering based on semantic features. On the other hand, MLLMs like LLaVA ( 27 ; 26 ) leverage LLMs and visual instruction tuning, enabling user-defined clustering through modality alignment. However, naively using MLLMs for multiple clustering can face several challenges. Firstly, MLLMs tend to produce unstructured data ( 24 ) , making it difficult to link semantically similar outputs to specific clustering labels. Furthermore, the model’s response may offer a too general image description instead of strictly following provided instructions. A comprehensive discussion of these problems is presented in Section 4.3 . To mitigate these issues, we propose Embedding Selective Multiple Clustering (ESMC). As shown in Figure 2 , ESMC first derives internal embeddings of text prompt tokens to extract features. These embeddings are from hidden states of text tokens of the language model in MLLMs. As visualized in Figure 3 , color-related features are strongly related to the ‘color’ prompt token, while prompt-unrelated or image-unrelated tokens present low logits. We also employ a clustering head with pseudo-label learning to group these embeddings into well-separated clusters. Our contributions can be summarized as follows. • We reveal that the internal embeddings of text prompt tokens can serve as condensed representations of corresponding features and can be utilized as an effective basis for multiple clustering. • We demonstrate that a lightweight 2-layer MLP clustering head with pseudo label supervision achieves substantial improvements. • Extensive experiments and ablation studies indicate that the proposed approach provides user-defined, diverse, and high-quality clustering outcomes. 2 Related Work Multiple clustering Multiple clustering aims to discover various meaningful ways to group a dataset ( 47 ) . Traditional methods often achieve this by partitioning data based on its internal characteristics, such as constructing orthogonal subspaces ( 6 ; 33 ) or modeling independent mixture distributions ( 19 ; 41 ) , employing objective functions to improve both cluster quality and diversity ( 47 ) . Recent deep learning advancements have also offered improved solutions using techniques like autoencoders, muMeasuring statistical dependence with Hilbert-Schmidt normsitepyao2023augdmc to learn better representations for diverse clustering. For example, MCV ( 12 ) utilizes different pre-trained feature extractors as diverse views for the same data to construct multiple clusters. ENRC ( 30 ) finds multiple non-redundant clusters in the embedded space of the autoencoder. iMClusts ( 39 ) leverages multi-head attention to generate significant subspaces and Hilbert Schmidt Independence Criterion (HSIC) ( 11 ) to minimize their dependencies. Multi-Map and Multi-Sub utilize CLIP ( 37 ) embeddings and proxy learning to generate multiple clusters. Most previous methods cannot provide clusters based on arbitrary user-defined criteria. Figure 2: The ESMC framework consists of three key components: (i) the MLLM processes multimodal inputs (images and text prompts); (ii) target embeddings are selected from the language model’s hidden states corresponding to text tokens; and (iii) a lightweight clustering head is trained to enhance cluster accuracy. Multi-modal large language models MLLMs typically integrate three core components: a vision encoder for extracting visual features, a projection network to align visual and text modalities, and an autoregressive language backbone. Taking LLaVA-1.5 ( 26 ) as an example, its architecture comprises CLIP ( 37 ) as the vision encoder, a projection module, and Vicuna-v1.5 ( 49 ) as the language foundation. Given an input image I , the CLIP image encoder f_{vision\_encoder} will provide visual vectors. A mapping network f_{proj} is applied to project image features and text features to the same space and provides X_{visual} tokens: \mathbf{X}_{visual}=f_{proj}(f_{vision\_enc}(I))\in\mathbb{R}^{n\times d_{mllm}}, (1) where d_{mllm} is the dimension of MLLM embedding. The input prompt S is mapped into m tokens through the tokenizer f_{tokenizer} to word embeddings from f_{embed} : \textbf{X}_{text}=f_{embed}(f_{tokenizer}(S))\in\mathbb{R}^{m\times d_{mllm}}. (2) The concatenated modality embeddings \left[\mathbf{X}_{vis};\mathbf{X}_{text}\right] are fed into the transformer-based language model. Let A_{l}(k)\in\mathbb{R}^{d_{model}} denote the k -th token’s representation at the l -th transformer layer, and A_{0}=\left[\mathbf{X}_{vis};\mathbf{X}_{text}\right] . The final output is generated from A_{L}\in\mathbb{R}^{{(m+n+t)}\times d_{model}} through an unembedding matrix \mathbf{W}u\in\mathbb{R}^{d{model}\times|V|} , where |V| is the vocabulary size to map the embeddings to the vocabulary space, and t represents special tokens. Cracking the Internal Mechanisms of MLLMs Detecting and understanding the internal workings of Large Language Models (LLMs) has long been recognized as crucial for trustworthy AI ( 7 ; 50 ; 48 ) . Inspired by these works, initial efforts have explored the interpretability of Multi-modal Large Language Models (MLLMs). For instance, Vl-interp ( 21 ) employs logit lens to identify and edit object hallucinations. LLaVA-interp ( 31 ) demonstrates a strong spatial correlation between object information and its original image location, indicating a refinement of visual input into interpretable language tokens. LVLM-interp ( 3 ) introduces interactive tools such as attention maps and causal graphs to aid