UniNote: A Unified Embedding Model for Multimodal Representation and Ranking Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of Korea DOI: 10.1145/3770855.3818503 ISBN: 979-8-4007-2259-2/2026/08 CCS: Computing methodologies Semantic networks CCS: Computing methodologies Visual content-based indexing and retrieval CCS: Information systems Similarity measures
Jinghan Zhao, Wenwei Jin, Anqi Li, Jintao Tong, Luya Mo, Jiawei Li, Bin Li, Yao Hu · 2026
Read UniNote: A Unified Embedding Model for Multimodal Representation and Ranking Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of Korea DOI: 10.1145/3770855.3818503 ISBN: 979-8-4007-2259-2/2026/08 CCS: Computing methodologies Semantic networks CCS: Computing methodologies Visual content-based indexing and retrieval CCS: Information systems Similarity measures by Jinghan Zhao, Wenwei Jin, Anqi Li, Jintao Tong, Luya Mo, Jiawei Li, Bin Li, Yao Hu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29287.txt.
Introduction
Jinghan Zhao and colleagues present UniNote, a unified embedding model for industrial item-to-item retrieval. Such retrieval underpins recommendation and auditing, yet current methods often struggle to balance global representation with fine-grained local matching. UniNote answers with tailored strategies for content at many levels of detail. Training runs in two stages: contrastive fine-tuning builds base embeddings, then reinforcement learning refines ranking quality. Deployed at Xiaohongshu, it reached state-of-the-art results with better quality and cost efficiency.
Catalog record
| CREATOR | Jinghan Zhao, Wenwei Jin, Anqi Li, Jintao Tong, Luya Mo, Jiawei Li, Bin Li, Yao Hu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44555 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29287
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UniNote: A Unified Embedding Model for Multimodal Representation and Ranking Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of Korea DOI: 10.1145/3770855.3818503 ISBN: 979-8-4007-2259-2/2026/08 CCS: Computing methodologies Semantic networks CCS: Computing methodologies Visual content-based indexing and retrieval CCS: Information systems Similarity measures Jinghan Zhao, Wenwei Jin, Anqi Li, Jintao Tong, Luya Mo, Jiawei Li, Bin Li, Yao Hu Abstract . Item-to-Item (I2I) retrieval is a fundamental part of modern content platforms, supporting critical industrial workflows from recommendation engines to content auditing. While multimodal embedding methods have advanced general retrieval, they often falter in I2I scenarios due to the challenges of balancing global content representation with fine-grained local retrieval, the systemic inefficiency of decoupled embedding-and-ranking pipelines, and the inherent trade-offs between model precision and serving latency. To solve these issues, we propose UniNote , a unified embedding model designed for industrial I2I retrieval. Tailored retrieval strategies are introduced to support representation learning over complex, multimodal content at varying granularities. To operationalize these strategies, UniNote employs a two-stage training paradigm: the first stage leverages contrastive SFT to establish robust base embeddings, while the second stage refines ranking quality through a reinforcement learning (RL) process that aligns the model with content relevance. Our results show that UniNote achieves SOTA performance across diverse I2I tasks. Deployed at Xiaohongshu and integrated with Matryoshka Representation Learning (MRL), UniNote achieved significant improvements in retrieval quality and cost efficiency in large-scale applications. † † cc-license: by 1. Introduction Figure 1. We propose UniNote, a unified retrieval-ranking framework specifically designed for I2I retrieval, which is capable of encoding various modality inputs into a unified space with cross-modal alignment (task: Atomic-level Alignment, OCR Perception), localized retrieval (task: Semantic Extraction, Subordinate Retrieval), and relevance-aware ranking (task: Content Relevance) capabilities. I2I retrieval constitutes a fundamental paradigm powering large-scale ecosystems in modern internet content platforms. As content modalities diversify and information volumes surge, retrieval requirements have evolved to be more multifaceted and demanding. Contemporary retrieval frameworks must adapt to heterogeneous inputs across various modalities, including text, images, videos, and their complex combinations, to support a broad range of applications such as recommendation systems ( 17 ; 23 ; 24 ; 25 ) , content moderation ( 27 ; 12 ; 1 ) , and data management. User-generated notes on the Xiaohongshu platform serve as a representative example of complex items that typically encompass heterogeneous modalities including text, images, videos. The retrieval of such notes entails several critical requirements. Effective retrieval of these notes imposes distinct functional requirements. First, the system must integrate multimodal information of each note into a cohesive global representation. Second, it necessitates fine-grained local retrieval capabilities, enabling tasks such as locating a full note based solely on an individual image. Third, the architecture is required to incorporate accurate relevance ranking while strictly adhering to the latency constraints of massive industrial databases. Despite the progress in multimodal embedding methods, addressing these requirements remains challenging due to three fundamental limitations in existing paradigms. (1) Structural disconnect in dual-tower models. Standard retrieval architectures ( e.g., CLIP ( 21 ) and its variants ( 28 ) ( 13 ) ) encode modalities independently to ensure efficiency. However, this late-interaction mechanism inherently fails to capture the fine-grained alignment between text and visual components required for composite notes, leading to a loss of local detail. (2) Granularity mismatch in generative baselines. While multimodal large language model (MLLM)-based embeddings ( 11 ) ( 30 ) excel at global reasoning, they suffer from a granularity mismatch. These models are optimized for broad semantic descriptions or question-answering tasks, often overlooking the hierarchical structure needed to retrieve a whole note based on a partial input ( i.e., local-to-global retrieval). (3) Inefficiency in relevance ranking. Regarding ranking, current industrial pipelines typically rely on a disjointed "retrieve-then-rerank" strategy ( 5 ) ( 14 ) . They use heavy generative models to rerank candidates, creating a significant computational bottleneck. Furthermore, models like NoteLLM-2 ( 29 ) , trained on user interaction data (clicks/likes), optimize for user interest rather than objective content relevance, making them ill-suited for strict semantic retrieval tasks where accuracy and consistency are paramount. To address these limitations, we propose UniNote , a unified embedding model tailored for the complex demands of I2I retrieval as illustrated in Figure 1 . UniNote bridges the gap between holistic encoding and fine-grained retrieval through a novel two-stage training paradigm. First, we introduce a Contrastive Supervised Fine-tuning stage. Unlike rigid dual-tower models, we develop a robust embedding model that compresses heterogeneous inputs into a unified vector space while supporting various I2I retrieval types. We introduce a suite of tailored retrieval settings to facilitate data-driven representation learning for complex multimodal content. This stage transforms an MLLM from a generative model into an embedding model by utilizing the last token as the final representation. Second, to overcome the ranking limitations of standard contrastive loss, we implement a Relative Reranking via Reinforcement Learning stage utilizing Group Relative Policy Optimization (GRPO) ( 22 ) . By constructing a hierarchical reward function that accounts for ranking positions and hard negatives, we directly optimize the embedding space for list-wise relevance. This enables the model to distinguish subtle content nuances without relying on computationally expensive generative rerankers, achieving high-precision ranking in a single pass. The main contributions are summarized as follows: • Comprehensive I2I Task Suite: We define ten distinct retrieval categories that span cross-modal alignment, information compression, and fine-grained local–global bidirectional retrieval. To support these, we construct a high-quality dataset with a robust hard-negative mining strategy. • Unified Retrieval-Ranking Framework: We propose UniNote , a unified model that bridges the gap between retrieval (embedding) and ranking (relevance). Through a novel two-stage training paradigm(Contrastive SFT followed by Relevance-aware Reinforcement Learning (via GRPO) ), we enable a single embedding to capture both coarse semantic spaces and fine-grained relevance. • SOTA Performance & Empirical Validation: Extensive evaluations demonstrate that UniNote achieves state-of-the-art results across all ten I2I tasks. Ablation studies confirm that the two training stages synergize to balance embedding and ranking requirements in retrieval tasks. • Industrial Deployment Support & Validation: To meet large-scale industrial requirements, we adopt the MRL embedding scheme, allowing dimension selection based on cost–effectiveness trade-offs for flexible deployment. Production evaluations demonstrate that this approach maintains superior retrieval performance under practical constraints. 2. Related Works 2.1. Multimodal Embedding Multimodal embedding aims to project heterogeneous data into a unified latent space. Early works, such as CLIP ( 21 ) , BLIP ( 13 ) , and SigLIP ( 28 ) , pioneered the use of large-scale image-text pairs coupled with contrastive learning to achieve cross-modal alignment. These methods typically adopt a dual-tower architecture, where independent encoders project images and text into a shared manifold. While these models exhibit remarkable zero-shot generalization and have become the backbone for various downstream tasks ( 3 ; 6 ; 7 ; 32 ; 33 ; 4 ; 20 ; 15 ; 16 ; 10 ) , their structural paradigm inherently limits them to single-modality inputs and often struggle