Hubness Reduction with Dual Bank Sinkhorn Normalization for Cross-Modal Retrieval Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dub

Zhengxin Pan, Haishuai Wang, Fangyu Wu, Peng Zhang, Jiajun Bu · 2025

Read Hubness Reduction with Dual Bank Sinkhorn Normalization for Cross-Modal Retrieval Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dub by Zhengxin Pan, Haishuai Wang, Fangyu Wu, Peng Zhang, Jiajun Bu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02538.txt.

Introduction

Zhengxin Pan and co-authors fix a popularity bias that skews search across media types, in a 2025 paper. In cross-modal retrieval, a few target items become hubs, turning up as the nearest match for query after query and drowning the right answers. The team first explains why a popular fix, inverted softmax, works: it balances how often targets get picked.

Their framework balances both sides, targets and queries, using a normalization borrowed from transport math. Since the true query pattern is unknown in practice, they add a target bank beside the query bank to close the gap between the stand-in and the real thing. Tests on image, video, and audio search against text all improved.

Catalog record

CREATORZhengxin Pan, Haishuai Wang, Fangyu Wu, Peng Zhang, Jiajun Bu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02538

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hubness Reduction with Dual Bank Sinkhorn Normalization for Cross-Modal Retrieval Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3755468 ISBN: 979-8-4007-2035-2/2025/10 3958 CCS: Information systems Video search Zhengxin Pan, Haishuai Wang, Fangyu Wu, Peng Zhang, Jiajun Bu Abstract . The past decade has witnessed rapid advancements in cross-modal retrieval, with significant progress made in accurately measuring the similarity between cross-modal pairs. However, the persistent hubness problem, a phenomenon where a small number of targets frequently appear as nearest neighbors to numerous queries, continues to hinder the precision of similarity measurements. Despite several proposed methods to reduce hubness, their underlying mechanisms remain poorly understood. To bridge this gap, we analyze the widely-adopted Inverted Softmax approach and demonstrate its effectiveness in balancing target probabilities during retrieval. Building on these insights, we propose a probability-balancing framework for more effective hubness reduction. We contend that balancing target probabilities alone is inadequate and, therefore, extend the framework to balance both query and target probabilities by introducing Sinkhorn Normalization (SN). Notably, we extend SN to scenarios where the true query distribution is unknown, showing that current methods, which rely solely on a query bank to estimate target hubness, produce suboptimal results due to a significant distributional gap between the query bank and targets. To mitigate this issue, we introduce Dual Bank Sinkhorn Normalization (DBSN), incorporating a corresponding target bank alongside the query bank to narrow this distributional gap. Our comprehensive evaluation across various cross-modal retrieval tasks, including image-text retrieval, video-text retrieval, and audio-text retrieval, demonstrates consistent performance improvements, validating the effectiveness of both SN and DBSN. All codes are publicly available at https://github.com/ppanzx/DBSN . Figure 1 . Retrieval comparisons under different query conditions. Left: SN vs current methods in query-aware scenarios (directly taking testing queries as query bank); Right: DBSN vs current methods in query-agnostic scenarios (leveraging training queries as query bank). Quantitative results can be found in § 4 and our Appendix. 1. Introduction Cross-modal retrieval involves identifying the nearest target from a target gallery in one modality based on a query from another modality. The primary challenge lies in accurately measuring the similarity between cross-modal pairs, which necessitates bridging both the heterogeneous and semantic gaps. Significant progress has been made over the past decade, particularly with advancements in visual-language pre-training ( Radford et al. 2021 ; Li et al. 2021 ; Jia et al. 2021 ; Li et al. 2022 ) . Despite these developments, the hubness problem, a critical issue that undermines the precision of similarity measurements and is pervasive in current cross-modal retrieval methods, remains unresolved. The hubness problem refers to a phenomenon where a small subset of hub targets frequently emerge as nearest neighbors to numerous queries, whereas some non-hub targets are rarely selected during retrieval ( Radovanovic et al. 2010 ) , as depicted in Figure 2 (a). This problem arises from the spatial centrality ( Hara et al. 2016 ) and the asymmetric nearest neighbor relations ( Schnitzer et al. 2012 ) . To mitigate hubness, existing methods fall into two paradigms: centering and scaling. Centering approaches address spatial centrality, whereas scaling approaches correct asymmetric relations. In cross-modal retrieval, scaling methods have demonstrated superior performance over centering methods ( Bogolin et al. 2022 ) ; however, their underlying mechanisms remain poorly understood. In this work, we analyze Inverted Softmax (IS) ( Smith et al. 2017 ) , a widely adopted scaling approach, to investigate its operational dynamics. Our analysis reveals that IS effectively balances the retrieval probabilities of targets. Inspired by this insight, we propose a probability-balancing framework for hubness reduction, which replaces the original similarity matrix with a doubly stochastic matrix that ensures uniform retrieval probabilities across all targets. In this framework, we establish that IS can be formulated as a specialized instance of balancing target probabilities through injecting hubness-compensation terms on the target dimension. However, we reveal that solely balancing target probabilities while leaving query probabilities unconstrained is fundamentally limited. Neglecting the query distribution may introduce systematic bias in hubness estimation. To address this limitation, we propose to simultaneously balance both query and target distributions, seeking to derive a doubly stochastic matrix. Within our probability-balancing framework, this objective corresponds to solving an entropy-constrained optimal transport problem. We leverage the Sinkhorn-Knopp algorithm ( Cuturi 2013 ) to obtain the solution, thus terming this approach Sinkhorn Normalization (SN). Consistent with IS, we prove that the core mechanism of SN functions through dual hubness compensation on queries and targets, enforcing balanced joint probabilities. We further investigate the efficacy of SN in query-agnostic scenarios, where only a single query is available under unknown distribution conditions, emulating real-world search engine deployments. An intuitive solution for these cases involves constructing a query bank to approximate the ground-truth query distribution for target hubness estimation ( Bogolin et al. 2022 ) . However, composing such a query bank is not only laborious but also not effective enough due to the nontrivial query-target distribution gap between the query bank and targets. DualIS ( Wang et al. 2023 ) attempts to bridge this gap via a target bank, yet yields limited improvements as it amplifies the query-query divergence between ground-truth queries and the augmented bank. Our proposed Dual Bank Sinkhorn Normalization (DBSN) strategically integrates the target bank on the target side instead of the query side, effectively reducing the query-target gap without enlarging the query-query gap. As shown in Figure 1 , consistent improvements across various retrieval tasks demonstrate the effectiveness of our SN and DBSN. Our contributions are summarized as follows: (1) We propose a probability-balancing framework for hubness reduction in cross-modal retrieval. Our framework reveals that IS operates by balancing target probabilities, thereby mitigating the hubness problem. (2) Within this framework, we identify the limitation of balancing target probabilities exclusively and resolve this through joint balancing query and target probabilities. To this end, we introduce Sinkhorn Normalization (SN). (3) We extend SN to query-agnostic scenarios and demonstrate that single-bank SN is suboptimal due to a significant query-target gap. We further propose Dual Bank Sinkhorn Normalization (DBSN), enhancing single-bank SN by narrowing the query-target gap with a target bank. Figure 2 . Structural comparisons between our SN/DBSN and current methods: (a) The hubness problem where the hub ( t_{1} ) is the nearest neighbor of multiple queries( q_{1} , q_{2} ), compromising retrieval precision. (b) IS/SN alleviates the hubness problem through query bank normalization. (c) DualIS expands queries to narrow the query-target gap but enlarges the query-query gap. (d) DBSN narrows the query-target gap while preserving the query-query gap by expanding targets instead of queries. 2. Related Works 2.1. Cross-Modal Retrieval. In this work, we investigate cross-modal retrieval, which involves identifying the most relevant target from a target gallery in one modality based on queries from distinct modalities. The key challenge lies in precise cross-modal similarity computation, manifesting through three primary aspects: the heterogeneous gap, the semantic gap, and the hubness problem. Notably, the first two challenges have been largely mitigated by Visual Semantic Embedding (VSE) ( Frome et al. 2013 ) , which pioneers dual encoders to project raw multimodal data

Read the complete text