Multi-modal Dynamic Proxy Learning for Personalized Multiple Clustering
Jinfeng Xu, Zheyu Chen, Shuo Yang, Jinze Li, Ziyue Peng, Zewei Liu, Hewei Wang, Jiayi Zhang, Edith C. H. Ngai · 2025
Read Multi-modal Dynamic Proxy Learning for Personalized Multiple Clustering by Jinfeng Xu, Zheyu Chen, Shuo Yang, Jinze Li, Ziyue Peng, Zewei Liu, Hewei Wang, Jiayi Zhang, Edith C. H. Ngai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07274.txt.
Introduction
Jinfeng Xu and eight co-authors personalize multiple clustering, the task of finding several other meaningful groupings of the same data. Current methods generate every possible clustering and leave users to sift through them manually. Multi-modal attempts to use language guidance suffer static rigidity: predefined candidate words miss dataset-certain concepts, and fixed fusion rules ignore how feature interactions evolve. Their system, Multi-DProxy, learns textual proxies instead.
A gated cross-modal fusion module builds joint representations by adaptively modeling feature interactions. Proxy optimization runs under two constraints: user interest constraints keep semantics aligned with domain concepts, while concept constraints mine hard examples to sharpen cluster separation. A dynamic candidate manager refines the proxies using clustering feedback. Tests across multi-clustering benchmarks showed state-of-the-art results with notable margins.
Catalog record
| CREATOR | Jinfeng Xu, Zheyu Chen, Shuo Yang, Jinze Li, Ziyue Peng, Zewei Liu, Hewei Wang, Jiayi Zhang, Edith C. H. Ngai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07274
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-modal Dynamic Proxy Learning for Personalized Multiple Clustering Jinfeng Xu, Zheyu Chen, Shuo Yang, Jinze Li, Ziyue Peng, Zewei Liu, Hewei Wang, Jiayi Zhang, Edith C. H. Ngai Abstract Multiple clustering aims to discover diverse latent structures from different perspectives, yet existing methods generate exhaustive clusterings without discerning user interest, necessitating laborious manual screening. Current multi-modal solutions suffer from static semantic rigidity: predefined candidate words fail to adapt to dataset-specific concepts, and fixed fusion strategies ignore evolving feature interactions. To overcome these limitations, we propose Multi-DProxy, a novel multi-modal dynamic proxy learning framework that leverages cross-modal alignment through learnable textual proxies. Multi-DProxy introduces 1) gated cross-modal fusion that synthesizes discriminative joint representations by adaptively modeling feature interactions. 2) dual-constraint proxy optimization where user interest constraints enforce semantic consistency with domain concepts while concept constraints employ hard example mining to enhance cluster discrimination. 3) dynamic candidate management that refines textual proxies through iterative clustering feedback. Therefore, Multi-DProxy not only effectively captures a user’s interest through proxies but also enables the identification of relevant clusterings with greater precision. Extensive experiments demonstrate state-of-the-art performance with significant improvements over existing methods across a broad set of multi-clustering benchmarks. Introduction Clustering, a cornerstone of unsupervised learning, aims to uncover latent structures by grouping data based on intrinsic similarities. Traditional works rely on handcrafted features or monolithic representations ( 13 ; 15 ; 2 ; 3 ) , often failing to capture the multifaceted nature of real-world data. While deep clustering works ( 4 ; 16 ; 19 ; 18 ) have improved expressiveness, they typically produce a single partitioning, disregarding the inherent complexity of data that can be meaningfully grouped from diverse perspectives. This limitation spurred the development of multiple clustering ( 14 ; 22 ; 28 ) , which seeks diverse partitions revealing complementary structures. However, existing works generate exhaustive clusterings without discerning user interest, necessitating laborious manual screening to identify relevant groupings—a significant practical bottleneck. Multimodal information is flooding the Internet ( 24 ; 25 ) . Recent works leverage multi-modal models like CLIP ( 20 ) to align user interests (expressed as keywords, e.g., “color”) with visual representations. Recent works such as Multi-MaP ( 30 ) and Multi-Sub ( 29 ) employ proxy learning, where textual prompts guide the extraction of interest-biased embeddings. Despite promising results, these solutions exhibit critical limitations: • Static Semantic Rigidity: Predefined candidate words (e.g., “red”, “blue”, “green” for “color”) fail to adapt to dataset-specific concepts, leading to misalignment when LLMs’ suggestions mismatch ground-truth categories. • Inflexible Feature Fusion: Fixed fusion strategies (e.g., concatenation or simple averaging) ignore evolving feature interactions between modalities, yielding suboptimal joint representations. To overcome these deficiencies, we introduce Multi-DProxy, a novel Multi-modal Dynamic Proxy Learning framework that synergizes gated cross-modal fusion, adaptive textual proxies, and dynamic candidates to generate personalized clusterings aligned with user interest. Our core innovations address the limitations head-on: • Gated Cross-Modal Fusion: A hierarchical attention module with sigmoid-gated residuals dynamically recalibrates visual-textual interactions, prioritizing discriminative attributes through bidirectional feature modulation. • Dual-Constraint Proxy Optimization: We enforce semantic consistency via user interest constraints (aligning proxies with concept centroids) while enhancing cluster discrimination via concept constraints using contrastive learning on fused features and relevant proxies. This replaces rigid candidate sets with learnable, semantically grounded proxies. • Dynamic Candidate Management: An iterative feedback loop refines textual semantics by scoring candidates against evolving cluster centroids. This continuously adapts proxies to emergent data structures, mitigating static rigidity. Multi-DProxy not only precisely captures user interests but also enables efficient identification of relevant clustering. Theoretical analysis proves proxy stability under dynamic updates and elucidates how visual features gate textual representations to prioritize salient attributes during fusion. Extensive experiments on a broad set of multi-clustering benchmarks demonstrate state-of-the-art performance. Our contribution can be summarized as: • The first framework unifying learnable textual proxies, dynamic candidate refinement, and adaptive feature fusion for interest-aware multiple clustering. • A theoretically grounded dual-constraint mechanism ensuring semantic coherence and cluster discrimination. • We conduct extensive experiments on all publicly available multiple clustering tasks, which empirically demonstrate the superiority of the proposed Multi-DProxy in precisely capturing the user’s interest. Related Work Multiple clustering explores diverse data partitions from different perspectives, gaining increasing attention. Early methods rely on hand-crafted rules and representations. For example, COALA ( 1 ) generates new clusters using existing ones as a constraint, Hu et al. ( 8 ) maximized eigengap across subspaces, and Dang et al. ( 5 ) utilize an expectation-maximization framework to optimize mutual information. Recent approaches leverage learning-based techniques for better representations. For instance, ENRC ( 14 ) optimizes clustering objectives within a latent space learned by an auto-encoder, iMClusts ( 22 ) leverages auto-encoders and multi-head attention to learn diverse feature representations, and AugDMC ( 28 ) applies data augmentation to generate diverse image perspectives. However, it remains challenging to identify the clustering most relevant to user interests. Recently, Multi-MaP ( 30 ) and Multi-Sub ( 29 ) integrate CLIP embeddings with proxy learning to generate data representations aligned with user interests. While effective, these methods exhibit static semantic rigidity: predefined candidate words fail to adapt to dataset-specific concepts, fixed fusion strategies ignore evolving feature interactions, and CLIP inherently lacks deep contextual understanding for nuanced intent capture ( 30 ; 29 ) . To address these limitations, we propose Multi-DProxy, a multi-modal dynamic proxy learning framework. Unlike static methods, Multi-DProxy leverages learnable textual proxies optimized via dual constraints—semantic consistency via concept centroid alignment and cluster discrimination via hard example mining. Methodology Multi-DProxy introduces a novel dynamic proxy learning framework that generates personalized clusterings aligned with user intent through adaptive cross-modal alignment. Multi-DProxy transforms high-level concepts into learnable textual proxies that guide visual feature extraction. As illustrated in Figure 1 . Figure 1: Overview of the Multi-DProxy framework. The central pipeline illustrates the overall architecture, while the key components are detailed on both sides: (1) Dynamic Candidate Management updates candidate words every R epochs; (2) Gated Cross-modal Fusion integrates visual and textual representations; (3) Cross-modal Alignment reduces modality discrepancies; (4) Concept Discrimination Constraints enhance cluster separability; and (5) User Interest Constraints ensure alignment with domain-specific concepts. Multi-modal Pre-training First, we briefly review the training objective in CLIP as follows, and then describe the details of our Multi-DProxy method based on that. Given a set of image-text pairs as \left\{v_{i},t_{i}\right\}_{i=1}^{D} , where D is the total number of datasets, and v_{i} is an image and t_{i} is the corresponding text description, their vision and text representations can be obtained by two encoders as \mathbf{v}_{i}=f_{t}(v_{i})\in\mathbb{R}^{d} and \mathbf{t}_{i}=f_{t}(t_{i})\in\mathbb{R}^{d} , where \mathbf{v}_{i} and \mathbf{t}_{i} have the unit norm and d is latent dimmension. Multi-DProxy employs frozen pre-trained CLIP encoders ( f_{v}(\cdot) for vision and f_{t}(\cdot) for text). Moreover, a user-specified concept u (e.g., “color”) to refer to user interest. Base Proxy Initialization For each input image x_{i} , we generate an initial base proxy embedding by processing a unified placeholder token “*” using CLIP’s reference word embedding function: \mathbf{w}_{i}^{\prime}=f_{t}(\text{``*"})\in\mathbb{R}^{d} . We initialize and maintain D different proxies \mathbf{W}=\left\{\mathbf{w}_{1},\ldots,\mathbf{w}_{D}\right\} . Multi-DProxy optimizes adaptive proxy embeddings \mathbf{w}_{i} , and facilitate identifying relevant clustering through three interconnected components: • Gated Cross-Modal Fusion: synthesizes discriminative joint representations through adaptive feature interaction. • Dynamic Candidate