Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalization

Xiaohan Wang, Zhangtao Cheng, Ting Zhong, Leiting Chen, Fan Zhou · 2025

Read Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalization by Xiaohan Wang, Zhangtao Cheng, Ting Zhong, Leiting Chen, Fan Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20258.txt.

Introduction

Wang, Cheng, Zhong, Chen, and Zhou balance fast and slow senses in multi-modal learning. Their 2025 paper proposes MBCD for domain generalization. Models must work on unseen domains, not just training ones. Weight averaging usually helps by finding flat minima. In multi-modal setups it backfires early though.

Fast modalities dominate averaging before slow ones mature. The fused model then leans on one sense too much. MBCD starts with adaptive modality dropout in the student. Randomly muting strong senses curbs the early bias. A gradient consistency rule aligns branch learning signals. Uni-modal and fused gradients are pushed to agree. A weight-averaged teacher then distills fused knowledge back. Each single-sense branch receives cross-modal teaching. Convergence steers toward flatter, sturdier solutions. EPIC-Kitchens and HAC benchmarks tested the framework. Accuracy and robustness beat prior methods across settings. Balanced fusion survives domain shift better.

Catalog record

CREATORXiaohan Wang, Zhangtao Cheng, Ting Zhong, Leiting Chen, Fan Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-20258

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalization Xiaohan Wang, Zhangtao Cheng, Ting Zhong, Leiting Chen, Fan Zhou Abstract Weight Averaging (WA) has emerged as a powerful technique for enhancing generalization by promoting convergence to a flat loss landscape, which correlates with stronger out-of-distribution performance. However, applying WA directly to multi-modal domain generalization (MMDG) is challenging: differences in optimization speed across modalities lead WA to overfit to faster-converging ones in early stages, suppressing the contribution of slower yet complementary modalities, thereby hindering effective modality fusion and skewing the loss surface toward sharper, less generalizable minima. To address this issue, we propose MBCD, a unified collaborative distillation framework that retains WA’s flatness-inducing advantages while overcoming its shortcomings in multi-modal contexts. MBCD begins with adaptive modality dropout in the student model to curb early-stage bias toward dominant modalities. A gradient consistency constraint then aligns learning signals between uni-modal branches and the fused representation, encouraging coordinated and smoother optimization. Finally, a WA-based teacher conducts cross-modal distillation by transferring fused knowledge to each uni-modal branch, which strengthens cross-modal interactions and steer convergence toward flatter solutions. Extensive experiments on MMDG benchmarks show that MBCD consistently outperforms existing methods, achieving superior accuracy and robustness across diverse unseen domains. Code is released at https://github.com/xiaohanwang01/MBCD . Introduction Domain Generalization (DG) ( 43 ) aims to build robust machine learning models that can generalize reliably on unseen target domains, even when trained exclusively on data from a limited set of source domains. In practice, conventional supervised learning models often suffer severe performance degradation when deployed in real-world scenarios, such as autonomous driving ( 40 ; 52 ) , medical image diagnosis ( 51 ; 53 ) , action recognition ( 18 ) , and industrial fault diagnosis ( 37 ; 5 ) . To address these challenges, researchers employ a variety of techniques, including domain alignment ( 27 ; 26 ) , data augmentation ( 42 ; 50 ) , loss landscape flattening ( 17 ; 48 ) and weight averaging (WA) ( 23 ; 38 ) . With the growing prevalence of multiple modalities in modern AI systems, such as vision-language pairs ( 36 ) and audio-visual ( 6 ) streams, multi-modal domain generalization (MMDG) has become an increasingly important yet complex challenge. In MMDG, models must learn to align and fuse heterogeneous modalities in the presence of distribution shifts, which significantly amplifies the complexity of the generalization, especially as models often rely on redundancy ( 41 ) . Figure 1: Illustration of EMA’s limitations in MMDG. (a) Comparison on the EPIC-Kitchens test set: the uni-modal models are trained from scratch, while the multi-modal model uses post-hoc classifier training. (b) Performance under varying modality shifts induced by Gaussian noise with different variance (shift level); performance drops sharply as the dominant modality (video) is perturbed. (c) Loss landscape visualization: EMA converges to sharp, biased minima under distributional shifts, while our MBCD yields flatter minima, improved robustness. Recent work ( 3 ) has highlighted the importance of loss landscape geometry for generalization, particularly focusing on flat minima that offer better resilience to distribution shifts. Among them, Exponential Moving Average (EMA) ( 28 ) stands out as a straightforward and powerful method: by averaging model parameters over training iterations, it smooths the optimization path and encourages convergence to flatter regions, while boosting out-of-distribution performance in uni-modal DG tasks. However, directly adapting EMA to multi-modal settings introduces new challenges. The main issue stems from optimization imbalance: modalities converge at different rates due to varying signal strengths and architectural. In multi-modal fusion networks, this causes EMA to disproportionately favor fast-converging (and often dominant) modalities during early training. Consequently, the contributions of slower yet complementary modalities are suppressed, hindering effective cross-modal fusion and weakening the representational capacity of the model. As shown in Figure 1 (a), the performance of the jointly trained model can even lag behind its uni-modal counterparts. This imbalance in optimization dynamics skews the averaged trajectory toward sharper, modality-biased minima (Figure 1 (c)), thereby impairing generalization. The vulnerability becomes more pronounced under dominant modality shifts, resulting in significant performance degradation, as illustrated in Figure 1 (b). Notably, the accuracy of the weaker modality like audio even increases under mild shifts, highlighting its limited contribution in the fused representation. To bridge this gap, we present M odality- B alanced C ollaborative D istillation (MBCD), a unified framework that retains EMA’s flatness benefits while explicitly mitigating its pitfalls in multi-modal learning. MBCD incorporate adaptive modality dropout into the student model to dynamically suppresses dominant modalities during training and reduce early-stage over-reliance. It then introduce a gradient consistency constraint to align the learning dynamics between uni-modal branches and the fused representation, enabling smoother, more coordinated optimization. Finally, an EMA-based teacher performs cross-modal distillation, injecting fused knowledge into each modality-specific branch, thereby enhancing cross-modal synergy and guiding the model toward flatter and more generalizable solutions. Overall, our contributions can be summarized as follows: • A detailed analysis of EMA’s core limitations in MMDG: optimization disparities lead to overfitting on dominant modalities, suppressing cross-modal interactions, and favoring sharper minima that harms generalization. • MBCD, a unified framework that maintains EMA’s strengths while alleviating modality imbalance through collaborative distillation, promoting balanced optimizations and flatter generalizations. • Comprehensive experiments on two public MMDG benchmarks demonstrate the effectiveness of our proposed MBCD across diverse modality combinations, yielding flatter minima and improved generalization. Related Work Domain Generalization. This task aims to train models on one or multiple source domains that can generalize to previously unseen target domains ( 57 ) . Specifically, domain alignment methods ( 27 ; 35 ) focus on learning domain-invariant representations. Moreover, data augmentation methods ( 54 ; 34 ; 56 ; 7 ) aim to simulate potential domain shifts by introducing variations at the input or feature level. Another line of research focuses on flattening the loss landscape ( 44 ; 48 ; 10 ) , encouraging convergence to flat minima. More recently, weight averaging methods ( 3 ; 2 ; 39 ; 38 ; 24 ) —such as stochastic weight averaging (SWA) ( 23 ) and exponential moving average (EMA) ( 31 ; 1 ) —have emerged as effective techniques for improving DG performance. Multi-Modal Domain Generalization. While uni-modal DG has been extensively studied, its multi-modal counterpart remains relatively underexplored. Existing efforts remain limited but provide promising directions. ( 33 ) align audio and visual features through a relative norm alignment loss to enhance cross-domain robustness and generalizable feature learning. ( 12 ) disentangles modality-specific and shared features with tailored constraints. ( 11 ) leverages self-supervised objectives like masked cross-modal translation and multimodal jigsaw puzzles to enhance MMDG. ( 14 ) promotes consistent flat minima and cross-modal knowledge transfer to enhance modality-specific learning. ( 22 ) proposes to mitigate modality asynchrony by learning a highly aligned and unified multi-modal representation space. However, prior approaches treat all modalities equally, resulting in weaker modalities being underoptimized and convergence to sharper minima. In contrast, our method suppresses dominant modalities, promoting wider minima and enhancing MMDG. Multi-Modal Imbalance Learning. A range of approaches ( 32 ; 15 ; 29 ) have been proposed to balance multi-modal learning ( 45 ) . For instance, ( 55 ) adopts alternating uni-modal

Read the complete text