GeoMM: On Geodesic Perspective for Multi-modal Learning
Shibin Mei, Hang Wang, Bingbing Ni · 2025
Read GeoMM: On Geodesic Perspective for Multi-modal Learning by Shibin Mei, Hang Wang, Bingbing Ni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11216.txt.
Introduction
Learning systems that mix images, text, and sound need a good way to judge how alike two samples are. Shibin Mei, Hang Wang, and Bingbing Ni argue in this 2025 paper that straight-line distance fails when samples sit on a curved space. Two items can look alike yet mean different things. They import a curved-space measure, geodesic distance, into multimodal learning.
In their method, named GeoMM, samples become nodes in a graph linked to their nearest neighbors, and distance is the shortest path along that graph. To keep the cost down, they group samples into clusters for a layered graph and update it bit by bit as data change. Tests across several downstream tasks showed gains over standard measures. The authors say the path-based distance better captures hidden links between samples.
Catalog record
| CREATOR | Shibin Mei, Hang Wang, Bingbing Ni |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11216
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GeoMM: On Geodesic Perspective for Multi-modal Learning Shibin Mei, Hang Wang, Bingbing Ni Abstract Geodesic distance serves as a reliable means of measuring distance in nonlinear spaces, and such nonlinear manifolds are prevalent in the current multimodal learning. In these scenarios, some samples may exhibit high similarity, yet they convey different semantics, making traditional distance metrics inadequate for distinguishing between positive and negative samples. This paper introduces geodesic distance as a novel distance metric in multi-modal learning for the first time, to mine correlations between samples, aiming to address the limitations of common distance metric. Our approach incorporates a comprehensive series of strategies to adapt geodesic distance for the current multimodal learning. Specifically, we construct a graph structure to represent the adjacency relationships among samples by thresholding distances between them and then apply the shortest-path algorithm to obtain geodesic distance within this graph. To facilitate efficient computation, we further propose a hierarchical graph structure through clustering and combined with incremental update strategies for dynamic status updates. Extensive experiments across various downstream tasks validate the effectiveness of our proposed method, demonstrating its capability to capture complex relationships between samples and improve the performance of multimodal learning models. 1 1 Introduction Recent years have witnessed remarkable progress on multimodal learning Li et al. 2019 ; Lu et al. 2019 ; Radford et al. 2021 ; Li et al. 2021a ; Bao et al. 2022 , driven by the rapid development of large transformer models Vaswani et al. 2017 . In multimodal learning, image-text pairs are encoded into a unified representation space through carefully designed pre-training tasks Chen et al. 2020c ; Li et al. 2020b ; Zhang et al. 2021 . The consistency of cross-modal representations facilitates the model with strong generalization abilities Yu et al. 2022 ; Wang et al. 2022 , achieving satisfactory performance on various downstream tasks. Figure 1 : The motivation of leveraging geodesic metric. Based on the topological structures, geodesic distance can depict sample similarity more accurately. A prevalent paradigm in multimodal learning includes extracting features from different modalities, encouraging proximity between matching features and dispersity between mismatched ones by contrastive supervision Radford et al. 2021 , such as InfoNCE loss He et al. 2020 . This paradigm inherently relies on distance computation between multi-modality features to mine positive and negative samples. CLIP Radford et al. 2021 computes cosine similarity between cross-modal samples within a large batch, while ALBEF Li et al. 2021a , maintains a momentum feature queue and calculates distances between batch features and queue features, thus avoiding large batch size. Recent works, such as TCL Yang et al. 2022a and MAFA Byun et al. 2024 , adopt a similar paradigm but pay more attention to negatives. However, these works assume that samples are distributed in a spherical space He et al. 2020 , overlooking the possibility of distributions in more complex non-Euclidean spaces. For example, some sentences similar at the word level may possess close distances with trivial metrics but convey vastly different semantics, as shown in Fig. 1 . We propose that geodesic distance Kimmel and Sethian 1998 can serve as an effective distance metric for multimodal learning, aiding in considering the data topology structure and thus more accurately depicting distances between samples. In this paper, we introduce geodesic distance to multimodal learning for the first time. Such distance metric enhances the characterization of relationships between samples in multi-modality datasets with complex data distribution. The benefits of using geodesic distance as a metric in multimodal learning are evident. Firstly, geodesic distance more accurately captures the underlying structure of complex data manifolds, especially when integrating data from diverse modalities. Therefore, we can easily distinguish samples with close cosine distances but distant geodesic distances to mine more accurate positive and negative samples. Secondly, traditional distance computations are one-to-one, meaning the distance between two points depends solely on those two points, which can lead to inaccuracy in portraying relative distances between distant points Chowdhury et al. 2022 . This ambiguity can adversely impact model optimization. In contrast, geodesic distance considers many-to-many (global) relationships, where data points are interconnected and the distance between two samples involves the participation of other routing points. The deterministic point-to-point path significantly reduces ambiguity and enhances interpretability in distance calculations, especially for more distant samples. Unlike traditional distance, the calculation of geodesic distance relies on a sample pool, which provides a reference frame for modeling distances between sample pairs. We posit that the local spaces neighboring each sample adhere to simple manifold assumption (as Def. 2 ), allowing us to compute distances using a straightforward one-to-one metric (as Def. 1 ). Consequently, we build a graph based on short-distance adjacency relationships within the sample pool, establishing undirected edges between locally adjacent samples. The distance between samples is represented as the shortest path between points in this graph, which can be computed using shortest-path algorithms. For simplicity, we employ the Floyd algorithm Floyd 1962 . However, directly calculating geodesic distance for a large number of samples would result in huge computational complexity. To facilitate efficient computation, we model the sample pool as a hierarchical graph. Within each layer of the graph, we partition samples into disjoint clusters through sample clustering. We then construct a graph among cluster centers and use the Floyd algorithm to calculate the shortest path between these cluster centers. Distances within each cluster are recursively propagated to the next layer of the hierarchical graph. At the bottom layer, we assume that the local manifold is simple enough to employ trivial distance metrics. For the distance between any two samples, we backtrack layer by layer in the hierarchical graph from bottom to top, to assess whether they belong to the same cluster, up to the top layer, as illustrated in Fig. 2 . Additionally, we develop efficient incremental update strategies to accommodate sample updates and reduce computational complexity. We believe geodesic distance can effectively facilitate current multimodal learning, yielding improved pre-trained models with more accurate distance modeling. Our multimodal pre-training experiments yield promising results across various tasks, such as image-text retrieval, visual question answering, visual entailment, natural language for visual reasoning, and visual grounding. Extensive results verify the effectiveness of this geodesic perspective. 2 Related Work 2.1 Multi-modal Learning Multimodal pre-training can be mainly categorized into single-stream networks Li et al. 2019 ; Lu et al. 2019 ; Su et al. 2019 ; Chen et al. 2020c ; Li et al. 2020b ; Huang et al. 2020b , dual-stream networks Tan and Bansal 2019 ; Radford et al. 2021 ; Li et al. 2021a ; Jia et al. 2021 ; Li et al. 2023b , and Xformer networks Yu et al. 2022 ; Li et al. 2022a ; Bao et al. 2022 ; Mei et al. 2025 ; Chen et al. 2021 . Single-stream networks concatenate text tokens and visual tokens and then utilize a transformer Vaswani et al. 2017 for modality fusion. This structure can be seen as treating visual modality as prompt for the text modality Li et al. 2023a . Depending on the way of obtaining visual tokens, single-stream networks can further be divided into region-based methods Chen et al. 2020c ; Li et al. 2020b ; Zhang