MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularization Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea. Proceedings of the 34th ACM International Conference on Information
Haochen You, Baojing Liu · 2025
Read MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularization Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea. Proceedings of the 34th ACM International Conference on Information by Haochen You, Baojing Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12149.txt.
Introduction
Haochen You and Baojing Liu present MOVER, a way to align text, video, and sound in one shared space for search across types. Their paper appeared in twenty twenty-five. Pair-wise drills work well for two types but scale poorly to three or more. They can also leave the shared space with weak shape and sense.
MOVER pairs a soft match rule drawn from transport math with a shape rule that shrinks the bulk of the point cloud. The shape rule, called GAVE, keeps groups tight and well spread by sense. Tests on text, video, and sound search beat top past methods in both raw and tuned runs. New type mixes also held up well.
Catalog record
| CREATOR | Haochen You, Baojing Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-12149
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularization Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea. Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM ’25), November 10–14, 2025, Seoul, Republic of Korea ISBN: xxxxxx DOI: xxxxxx CCS: Computing methodologies Machine learning Haochen You, Baojing Liu Abstract . Recent advances in multimodal learning have largely relied on pairwise contrastive objectives to align different modalities, such as text, video, and audio, in a shared embedding space. While effective in bi-modal setups, these approaches struggle to generalize across multiple modalities and often lack semantic structure in high-dimensional spaces. In this paper, we propose MOVER , a novel framework that combines optimal transport-based soft alignment with volume-based geometric regularization to build semantically aligned and structured multimodal representations. By integrating a transport-guided matching mechanism with a geometric volume minimization objective (GAVE), MOVER encourages consistent alignment across all modalities in a modality-agnostic manner. Experiments on text-video-audio retrieval tasks demonstrate that MOVER significantly outperforms prior state-of-the-art methods in both zero-shot and finetuned settings. Additional analysis shows improved generalization to unseen modality combinations and stronger structural consistency in the learned embedding space. 1. Introduction With the rapid development of foundation models, multimodal learning has become a cornerstone for building general-purpose intelligent systems capable of understanding and generating content across diverse modalities such as vision, audio, and language ( Zheng et al. 2024 ) . A central goal in this field is to construct a unified embedding space where semantically related data from different modalities are aligned ( Wu et al. 2024 ) , enabling tasks such as cross-modal retrieval, captioning, and audiovisual grounding ( Zong et al. 2023 ) . To this end, most existing approaches adopt contrastive learning frameworks that align pairs of modalities-typically using text as a common anchor-and optimize pairwise similarity in a latent space ( Li et al. 2024 ; You and Liu 2024 ) . Figure 1. Overview of the MOVER framework. Given inputs from k modalities, modality-specific encoders produce unit-normalized embeddings, which are soft-matched via cross-modal optimal transport. GAVE module computes the geometric volume of matched groups as a measure of alignment. The final loss combines transport-based weighting and volume alignment to guide multimodal representation learning. While effective in bi-modal settings, pairwise strategies face clear limitations when extended to three or more modalities ( Cicchetti et al. 2024 ) . They often rely on fixed anchors or manual fusion, which may disrupt mutual alignment across modalities ( Chen and Huang 2025 ; Zhang et al. 2024b ) . Moreover, similarity metrics like cosine similarity capture only local pairwise relations, ignoring global structure ( Qian et al. 2025 ) . This can lead to semantic inconsistency, modality dominance ( Yu et al. 2024 ) , and entangled representations that hinder generalization and interpretability ( Zhu and Luo 2024 ) . To address these challenges, recent efforts have explored geometric alignment mechanisms beyond simple pairwise matching ( Wang and Shen 2025 ; Tjandrasuwita et al. 2025 ) . For instance, volume-based objectives have been proposed to ensure embeddings from multiple modalities form coherent and semantically meaningful structures in latent space ( Cicchetti et al. 2024 ; Ren and Li 2023 ) . While these approaches mark an important step forward, they still face two key limitations: their alignment mechanisms are typically based on hard matching or static pairing, lacking flexibility in modeling global semantic structures ( Yuan et al. 2025 ) ; moreover, it remains underexplored how to construct a principled and transferable alignment criterion in the absence of direct supervision for each modality pair ( Zhang et al. 2024a ) . In this paper, we propose MOVER , a new framework for multimodal representation learning that integrates optimal transport-based soft alignment with volume-based geometric regularization. By combining flexible many-to-many matching with compact volume preservation, MOVER promotes semantically consistent and structure-aware representations across modalities. Our main contributions are as follows: • We introduce MOVER, a unified framework that leverages optimal transport and volume-based alignment (GAVE) for structured multimodal representation learning (Figure 1 ). • We design an end-to-end contrastive objective that jointly enhances semantic discriminability and modality alignment, applicable in both zero-shot and finetuned regimes. • We demonstrate state-of-the-art performance on multiple benchmarks, including text-video and text-audio retrieval, and validate the effectiveness of MOVER through ablations and visualization. 2. Method 2.1. Multimodal Embedding Module (Encoders) We consider a k -modal representation learning setting, where the m -th modality corresponds to a data space \mathcal{X}^{(m)} for m=1,\ldots,k . Given a batch of multimodal samples, each sample consists of inputs from all k modalities: (1) \left(x_{i}^{(1)},x_{i}^{(2)},\ldots,x_{i}^{(k)}\right),\quad i=1,\ldots,B, where x_{i}^{(m)}\in\mathcal{X}^{(m)} denotes the raw input from the m -th modality for the i -th sample, and B is the batch size. For each modality m\in\{1,2,\ldots,k\} , we define a modality-specific encoder E^{(m)}:\mathcal{X}^{(m)}\rightarrow\mathbb{R}^{d} that maps the input into a d -dimensional embedding vector. All outputs are L_{2} -normalized to ensure cross-modal comparability: (2) \mathbf{v}_{i}^{(m)}=E^{(m)}\left(x_{i}^{(m)}\right)\big/\left\|E^{(m)}\left(x_{i}^{(m)}\right)\right\|,\quad\forall i=1,\ldots,B. As a result, each sample i is represented by a set of k unit-normalized embeddings: (3) \left(\mathbf{v}_{i}^{(1)},\mathbf{v}_{i}^{(2)},\ldots,\mathbf{v}_{i}^{(k)}\right),\quad\mathbf{v}_{i}^{(m)}\in\mathbb{S}^{d-1}, where \mathbb{S}^{d-1} denotes the unit sphere in \mathbb{R}^{d} . This normalization ensures consistent scaling across modalities and stabilizes volume computation and optimal transport. Each encoder E^{(m)} adopts an architecture suited to its modality, and all encoders are jointly optimized using the unified alignment loss to learn semantically aligned representations in a shared space. 2.2. Modality Matching Module After obtaining unit-normalized embeddings from all k modalities, the next goal is to identify semantically aligned cross-modal sample groups, even when explicit alignment is not provided in the original data. We introduce an optimal transport (OT) mechanism to compute probabilistic matchings between modalities, producing soft-aligned candidate groups for subsequent geometric alignment. Let \mathcal{V}^{(m)}=\left\{\mathbf{v}_{1}^{(m)},\ldots,\mathbf{v}_{B}^{(m)}\right\} denote unit-norm embeddings for the m -th modality in a batch, where \mathbf{v}_{i}^{(m)}\in\mathbb{S}^{d-1} . For each pair of modalities p,q\in\{1,\ldots,k\} , we define a cost matrix C^{(p,q)}\in\mathbb{R}^{B\times B} based on the squared Euclidean distance between embeddings: (4) C^{(p,q)}[i,j]=\left\|\mathbf{v}_{i}^{(p)}-\mathbf{v}_{j}^{(q)}\right\|^{2}. This cost matrix reflects the semantic discrepancy between samples across modalities. Given the cost matrix, we apply entropic regularized optimal transport (e.g., Sinkhorn algorithm) to solve for a soft matching matrix: (5) \pi^{(p,q)}=\operatorname{Sinkhorn}\left(\mu^{(p)},\mu^{(q)},C^{(p,q)},\varepsilon\right), where \pi^{(p,q)}\in\mathbb{R}^{B\times B} denotes the soft matching weights from modality p to q , \mu^{(p)} and \mu^{(q)} are uniform marginals over samples in p and q , and \varepsilon is an entropic regularization coefficient. This process is repeated for all modality pairs, resulting in a set of soft matching matrices \left\{\pi^{(p,q)}\right\}_{1\leq p<q\leq k} defining probabilistic alignment between samples across modalities. Based on them, we generate candidate cross-modal groups using one of three strategies: hard matching , which selects the most probable match in each \pi^{(p,q)} for every sample; top- k^{\prime} matching , which retains the top- k^{\prime} matches per modality pair; and soft sampling , which samples indices according to the matching distributions. The resulting combinations are passed to the geometric alignment module, which evaluates structural consistency via volume computation. 2.3. Geometric Alignment Module In the framework of GAVE (Geometric Alignment via Volume Embedding), we measure cross-modal alignment by evaluating the geometric consistency of the modality embeddings in a shared latent space. Specifically, we compute the volume of the