e5-omni : Explicit Cross-modal Alignment for Omni-modal Embeddings
Haonan Chen, Sicheng Gao, Radu Timofte, Tetsuya Sakai, Zhicheng Dou · 2026
Read e5-omni : Explicit Cross-modal Alignment for Omni-modal Embeddings by Haonan Chen, Sicheng Gao, Radu Timofte, Tetsuya Sakai, Zhicheng Dou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03666.txt.
Introduction
Haonan Chen and four co-authors present e5-omni, a recipe for embedding models that place text, images, video, and audio in one shared space for direct comparison. Such omni-modal embeddings power search across mixed media. Most current models inherit their alignment quietly from a pretrained vision-language backbone, and the authors identify three recurring faults that follow from that shortcut.
Similarity scores land on inconsistent scales across modalities. In-batch negative examples grow too easy over time, weakening training. The embedding statistics of different modalities also mismatch, making rankings unstable. The e5-omni recipe fixes alignment explicitly while staying light, adapting off-the-shelf backbones into robust omni-modal models. Consistent cross-media scoring is basic infrastructure for search engines that must rank a photo against a paragraph fairly.
Catalog record
| CREATOR | Haonan Chen, Sicheng Gao, Radu Timofte, Tetsuya Sakai, Zhicheng Dou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03666
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
e5-omni : Explicit Cross-modal Alignment for Omni-modal Embeddings Haonan Chen, Sicheng Gao, Radu Timofte, Tetsuya Sakai, Zhicheng Dou Abstract Modern information systems often involve different types of items, e.g. , a text query, an image, a video clip, or an audio segment. This motivates omni-modal embedding models that map heterogeneous modalities into a shared space for direct comparison. However, most recent omni-modal embeddings still rely heavily on implicit alignment inherited from pretrained vision-language model (VLM) backbones. In practice, this causes three common issues: (i) similarity logits have modality-dependent sharpness, so scores are not on a consistent scale; (ii) in-batch negatives become less effective over time because mixed-modality batches create an imbalanced hardness distribution; as a result, many negatives quickly become trivial and contribute little gradient; and (iii) embeddings across modalities show mismatched first- and second-order statistics, which makes rankings less stable. To tackle these problems, we propose e5-omni , a lightweight explicit alignment recipe that adapts off-the-shelf VLMs into robust omni-modal embedding models. e5-omni combines three simple components: (1) modality-aware temperature calibration to align similarity scales, (2) a controllable negative curriculum with debiasing to focus on confusing negatives while reducing the impact of false negatives, and (3) batch whitening with covariance regularization to better match cross-modal geometry in the shared embedding space. Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines, and the same recipe also transfers well to other VLM backbones. We release our model checkpoint at https://huggingface.co/Haon-Chen/e5-omni-7B . 1 Introduction Modern information retrieval increasingly needs to handle multimodal queries and results in web search, multimedia question answering, and long-document understanding ( Jiang et al. 2025 ; Meng et al. 2025 ; Ma et al. 2024 ) . Recent vision–language models (VLMs) have shown strong multimodal capabilities, and bi-modal embedding models fine-tuned from VLMs perform well on text–image retrieval ( Chen et al. 2025b ; Zhang et al. 2024 ) . However, many real-world retrieval scenarios involve additional modalities beyond text and images, such as audio and video. Motivated by this, we study an omni-modal setting where heterogeneous items (text, images, audio, and video) are embedded into a shared space for consistent similarity comparison. Figure 1: Implicit vs. explicit alignment for omni-modal embeddings. (a) Implicit alignment leads to modality-dependent sharpness, negative hardness imbalance, and unstable ranking. (b) e5-omni performs explicit alignment with three lightweight modules to calibrate cross-modality similarities. Recent work has started to move from bi-modal to omni-modal embedding models, e.g., by adopting a VLM-centered bi-encoder architecture ( Ma et al. 2025 ; Xu et al. 2025b ; Xiao et al. 2025 ) . However, these methods still largely rely on implicit modality alignment inherited from the generative pretraining of VLMs. Contrastive fine-tuning provides only a limited corrective signal, and it may not fully calibrate modality-specific similarity scales or the local geometry of the shared space. As shown in the left part of Fig. 1 , omni-modal contrastive training is challenging because different modalities induce similarity scores with different scales, hardness profiles, and embedding geometry. This often leads to three issues: (1) Modality-dependent sharpness: using a single global temperature can produce overly sharp logits for some modality compositions but overly flat logits for others, which creates imbalanced contrastive gradients; (2) Negative hardness imbalance: negative hardness varies across modalities and shifts during training; when all in-batch negatives are treated equally, optimization can be dominated by many trivially easy negatives, weakening the late-stage learning signal and limiting fine-grained discrimination in mixed-modality batches; and (3) Unstable ranking: inconsistent geometry across heterogeneous inputs makes rankings sensitive to small score changes, even when all items are embedded in a single space. These issues become more significant in the omni-modal setting, because mini-batches naturally mix diverse modalities and thus amplify scale and geometry mismatch. In this work, we propose a lightweight explicit alignment method ( e5-omni ) that turns VLM backbones into robust omni-modal embedding models without modifying the architecture. As shown in the right part of Fig. 1 , e5-omni has three components: (1) Modality-aware Temperature Calibration. We introduce a lightweight calibration module that uses a trainable per-modality scaling vector to calibrate similarity logits across modalities. This adaptive rescaling helps balance contrastive training signals across different modality compositions. (2) Controllable Negative Curriculum. We select in-batch negatives using a quantile-based threshold and gradually increase hardness after a warmup period. To mitigate the bias introduced by negative selection, we further incorporate a debiased contrastive objective ( Chuang et al. 2020 ) , which stabilizes optimization in mixed-modality batches. (3) Batch Whitening and Covariance Alignment. We apply a batch whitening transform ( Ermolov et al. 2021 ) to the embeddings and add a CORAL-style regularizer ( Sun and Saenko 2016 ) to align mini-batch covariances across modalities. This harmonizes second-order geometry in the shared space and leads to more consistent similarity comparisons across diverse omni-modal inputs. Empirically, e5-omni delivers consistent improvements on diverse embedding benchmarks, including MMEB-V2 ( Meng et al. 2025 ) and the AudioCaps text–audio retrieval benchmark ( Kim et al. 2019 ) . Moreover, the gains grow with model scale: upgrading the backbone from 3B to 7B yields larger improvements. We also find that e5-omni transfers well to other VLM backbones. In summary, our contributions are threefold: • We propose e5-omni , a lightweight explicit alignment recipe that adapts generative VLMs into omni-modal embedding models without changing the backbone architecture. • We introduce three simple plug-in components: modality-aware logit calibration, a controllable negative curriculum with debiased contrastive learning, and batch whitening with covariance alignment, making omni-modal contrastive training more robust under mixed-modality batches. • We achieve consistent gains on MMEB-V2 and AudioCaps over strong baselines, and show that the recipe benefits more from larger backbones and transfers to other VLMs. 2 Method: e5-omni Figure 2: Overview of e5-omni . Given omni-modal inputs, e5-omni augments a VLM backbone with three lightweight components: (1) Modality-aware temperature calibration computes a modality-composition indicator w(x) and applies learnable modality-specific temperatures \boldsymbol{\tau} to calibrate logits \ell(q,p) ; (2) Controllable negative curriculum progressively masks easy negatives and optimizes a masked debiased objective \mathcal{L}_{\mathrm{DCL}} ; (3) Batch whitening and covariance alignment whitens batch embeddings and adds a CORAL-style covariance regularizer. In this section, we present e5-omni , a lightweight explicit-alignment framework that transforms a VLM into a unified omni-modal embedding model. As shown in the left part of Fig. 2 , e5-omni preserves the backbone and introduces a simple training recipe. Our framework comprises three components: (1) Modality-aware temperature calibration, which balances contrastive sharpness across modalities; (2) A controllable negative curriculum, which maintains a strong and stable learning signal; and (3) Batch whitening and covariance alignment, which harmonizes the shared-space geometry under heterogeneous inputs. Together, these components improve the robustness of omni-modal embeddings without architectural modifications. 2.1 Preliminaries We study omni-modal embedding with a base modality set \mathcal{M}_{0}=\{\texttt{T},\texttt{I},\texttt{A},\texttt{V}\} , corresponding to text , image , audio , and video . Training data consist of tuples (q,p^{+},\mathcal{P}^{-}) , where q is a query, p^{+} is its matched target, and \mathcal{P}^{-} is a set of negative targets. In this setting, q , p^{+} , and each p^{-}\in\mathcal{P}^{-} can each take any non-empty modality composition: q=x^{m_{q}},p^{+}=x^{m_{+}},p^{-}=x^{m_{-}},\quad m_{q},m_{+},m_{-}\in 2^{\mathcal{M}_{0}}\setminus\{\emptyset\} , where we denote an input with modality composition m as