MMDG-Bench: A Benchmark for Multimodal Domain Generalization
Qianshan Zhan, Qian Wang, Da Li, Xiao-Jun Zeng, Xiatian Zhu · 2026
Read MMDG-Bench: A Benchmark for Multimodal Domain Generalization by Qianshan Zhan, Qian Wang, Da Li, Xiao-Jun Zeng, Xiatian Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00891.txt.
Introduction
Zhan and co-authors set a shared test for models that must generalize across domains and senses in this 2026 paper. Multimodal domain generalization uses several senses, like video and audio, to stay robust on unseen domains. The two parent fields are mature on their own. Their union lacks standard tests, and most work sits in action recognition.
Their benchmark, MMDG-Bench, frames two build orders. One does domain generalization first, then multimodal learning. The other flips the order. Shared protocols cover action recognition with video, audio, and flow. They also cover face anti-spoofing with color, depth, and infrared views. Ten baselines pair one multimodal setup with five generalization tricks, in both orders. These plain builds often beat bespoke state-of-the-art methods, they report. Three lessons follow: generalization tricks help across backbones, and the best order turns on how stable cross-sense links are.
Catalog record
| CREATOR | Qianshan Zhan, Qian Wang, Da Li, Xiao-Jun Zeng, Xiatian Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00891
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMDG-Bench: A Benchmark for Multimodal Domain Generalization Qianshan Zhan, Qian Wang, Da Li, Xiao-Jun Zeng, Xiatian Zhu Abstract Multi-modal Domain Generalization (MMDG) seeks to leverage complementary modalities to enhance model robustness on unseen domains. Despite extensive progress in Multi-modal Learning (MML) and Domain Generalization (DG) as individual fields, their systematic integration remains under-explored. Current MMDG research is largely confined to action recognition and lacks standardized evaluation protocols. To address this, we introduce MMDG-Bench, a comprehensive benchmark featuring two foundational frameworks: “DG then MML” (D2M) and “MML then DG” (M2D). We provide unified experimental protocols across diverse tasks, including video-audio-flow action recognition and RGB-Depth-IR face anti-spoofing. By instantiating ten MMDG baselines through pairing a unified MML configuration with five DG techniques under both D2M and M2D orderings, we demonstrate that these structured combinations frequently outperform existing state-of-the-art methods, underscoring the necessity of a unified benchmarking effort. Our analysis yields three key insights: (1) Integrating DG techniques provides consistent generalization gains across various backbones, whereas non-DG methods are highly sensitive to backbone shifts; (2) The optimal framework choice depends on inter-modal stability: D2M excels when modal relations are stable across domains, while M2D is more robust to cross-domain relational variance; (3) Stronger backbones yield amplified performance dividends when integrated into our structured frameworks. MMDG-Bench provides a principled foundation and actionable design guidelines for future research in multi-modal robustness. Code is released at https://github.com/qszhan/MMDG-Bench . 1 Introduction Multi-modal domain generalization (MMDG) [ 5 ] leverages complementary information across modalities for robust generalization to unseen domains. MMDG addresses two interacting gaps: (1) the modality gap from heterogeneity across modalities, and (2) the domain gap from environmental and sensor variations. This interaction makes MMDG fundamentally more complex than single-modal domain generalization (DG) [ 13 ] or intra-domain multi-modal learning (MML) [ 38 ] . While MML and DG have been extensively studied, existing MMDG research lacks systematic investigation of leveraging and evaluating these methods and their combination. Early MMDG studies focused on modality alignment and fusion [ 31 , 5 , 6 , 37 , 16 ] while overlooking the extensive MML literature and typically merging source domains at training. More recent efforts incorporated domain concepts but overlooked DG literature [ 15 , 23 , 22 ] . Furthermore, prior MMDG work focused exclusively on action recognition and lacked standardized evaluation protocols across tasks. To address these limitations, we present MMDG-Bench, comprising two complementary frameworks: “DG then MML” (D2M) as illustrated in Fig. 1(a) and “MML then DG” (M2D) as in Fig. 1(b) , enabling systematic integration and evaluation of existing DG and MML methods. D2M learns domain-invariant representations per modality before cross-modal fusion; M2D aligns modalities then enhances domain generalization on fused features. We establish unified protocols for video-audio-flow action recognition and RGB-Depth-IR face anti-spoofing, implementing ten MMDG variants by pairing a unified MML configuration with five DG techniques. Our design is flexible and can accommodate additional methods and tasks. Extensive experiments validate our frameworks, with variants that often surpass prior state-of-the-art results. Our analysis reveals key design principles for framework selection, DG integration, and practical deployment. This work establishes a standardized foundation for MMDG research through unified frameworks, rigorous benchmarking, and actionable guidance. 2 Related Work Multi-modal Learning leverages complementary information across modalities to learn richer and more robust representations than single-modality models. A major line of work focuses on feature alignment, where contrastive objectives pull paired representations together while preserving modality-specific cues, as shown in CLIP [ 32 ] , ALIGN [ 17 ] , and translation-based models that reconstruct one modality from another to enforce cross-modal consistency [ 5 ] . Beyond representation learning, optimization-based balancing methods such as Gradient Blending [ 36 ] address modality dominance by adaptively reweighting gradients. Fusion-oriented approaches study how to combine modalities at different stages, from early fusion [ 30 ] to late fusion [ 39 ] , with attention bottlenecks [ 29 ] emerging as a compact and effective solution for cross-modal interaction. Domain Generalization aims to train models on multiple source domains that generalize to unseen targets [ 20 ] . DG approaches fall into three categories: data manipulation, representation learning, and learning strategy–based methods [ 34 ] . Data manipulation enhance generalization by increasing diversity through techniques like Mixup [ 42 ] , which interpolates samples and labels to encourage smoother decision boundaries. Representation learning seeks domain-invariant features by minimizing statistical or semantic discrepancies, as in MMD [ 12 ] or cross-domain contrastive learning [ 27 ] . Learning strategy–based methods improve generalization through meta-learning [ 21 ] and self-supervision [ 1 ] . Notably, MIRO [ 2 ] introduces a mutual-information regularization objective that enforces semantic consistency across domains, boosting generalization. Multi-modal Domain Generalization aims to learn from multi-modal source domains and generalize to unseen targets. Early studies primarily focused on MML. For action recognition, RNA-Net [ 31 ] aligned audio–visual norms to enhance cross-modal robustness, while SimMMDG [ 5 ] jointly learned shared and modality-specific features to improve alignment. CMRF [ 6 ] further flattened cross-modal manifolds to reduce modality bias. More recently, MBCD [ 37 ] mitigates modality imbalance induced by naïve weight averaging via collaborative distillation, and MAD-DG [ 16 ] tackles asynchronous modality misalignment through segment-label aligned temporal binding. These methods merge data from all source domains, implicitly assuming domain homogeneity and thus limiting generalization. More recent efforts explicitly incorporate DG into multi-modal learning. For action recognition, UR-Mixup [ 15 ] augments mixed-domain data. Li et al. [ 22 ] adversarially suppress domain-dependent patterns in both modality-specific and fused representations. For face anti-spoofing, Lin et al. [ 23 ] introduce domain-specific prototypes. Despite progress, existing approaches lack systematic frameworks integrating MML and DG, and most focus solely on action recognition without standardized protocols for fair comparison. 3 MMDG-Bench 3.1 Problem Formulation Let \mathcal{X}=\prod_{m=1}^{M}\mathcal{X}_{m} and \mathcal{Y} denote the M -modal input space and the output space, respectively. We are given a set of K source domains \mathcal{S}=\{D_{s}^{(k)}\}_{k=1}^{K} , where D_{s}^{(k)}=\{(\mathbf{x}_{i}^{(k)},y_{i}^{(k)})\}_{i=1}^{n_{k}} contains n_{k} labeled instances. Each instance \mathbf{x}=\{\mathbf{x}_{m}\}_{m=1}^{M} consists of M modalities. The domains are subject to distribution shifts, i.e., P_{XY}^{(i)}\neq P_{XY}^{(j)} for any i\neq j . The goal of MMDG is to learn a predictive function f:\mathcal{X}\rightarrow\mathcal{Y} that achieves minimum risk on an unseen target domain D_{t} : f=\arg\min_{f}\mathbb{E}_{(\mathbf{x},y)\sim D_{t}}\left[\ell(f(\mathbf{x}),y)\right] , where \ell(\cdot,\cdot) denotes the loss function. As D_{t} is inaccessible during training, MMDG seeks domain and modality invariance via a surrogate objective over \mathcal{S} . Let \{\mathbf{z}^{(k)}\}_{k=1}^{K} denote features grouped by domain and \{\mathbf{z}_{m}\}_{m=1}^{M} denote features grouped by modality. The objective is defined as \min_{f}\sum_{k=1}^{K}\mathbb{E}_{(\mathbf{x},y)\sim D_{s}^{(k)}}\left[\ell(f(\mathbf{x}),y)\right]+\lambda_{d}\Omega_{DG}\left(\{\mathbf{z}^{(k)}\}_{k=1}^{K}\right)+\lambda_{m}\Omega_{MML}\left(\{\mathbf{z}_{m}\}_{m=1}^{M}\right) , where the regularizer \Omega_{DG} and \Omega_{MML} enforce domain- and modality-invariance, and \lambda_{d},\lambda_{m}>0 weight their contributions. 3.2 D2M and M2D Frameworks (a) D2M: domain alignment first, modal fusion second. (b) M2D: modal fusion first, domain alignment second. Figure 1 : Two MMDG frameworks. Modal-specific encoders E_{v} and E_{a} extract features from domains D_{0} and D_{1} , with partially frozen backbones and trainable upper layers. MML aligns modalities via modality translation ( \mathcal{L}_{\text{mt}} ) and modality contrastive loss ( \mathcal{L}_{\text{mc}} ) before bottleneck fusion, while GBlend outputs modality-wise loss fusion weights \mathbf{w}^{T} . DG uses ERM ( \mathcal{L}_{\text{ERM}} ) with optional regularizers: Mixup ( \mathcal{L}_{\text{mix}} ), MMD ( \mathcal{L}_{\text{mmd}} ), contrastive regularization