Table 7:

Comparison using CMU-MOSEI dataset. · 2025

Read Table 7: by Comparison using CMU-MOSEI dataset. (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06686.txt.

Introduction

Heshan Fernando and co-authors rebalance training when a control system learns from several senses at once. Vision, audio, and language channels learn at different speeds, and the fast channels hog the optimization while slow ones stay undertrained, leaving the system fragile when a sensor is disturbed. Heuristic balancing tricks exist but can be costly and ad hoc.

Recasting the problem as multi-objective optimization lets the method explicitly favor the worst-performing modality while keeping the overall fusion goal. Their gradient method, MIMO, carries convergence guarantees. On standard multimodal benchmarks it improved balanced performance over leading rivals and cut subroutine computation time by up to about twentyfold.

Catalog record

CREATORComparison using CMU-MOSEI dataset.
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06686

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Table 7: Comparison using CMU-MOSEI dataset. Abstract Multi-modal learning (MML) aims to integrate information from multiple modalities, which is expected to lead to superior performance over single-modality learning. However, recent studies have shown that MML can underperform, even compared to single-modality approaches, due to imbalanced learning across modalities. Methods have been proposed to alleviate this imbalance issue using different heuristics, which often lead to computationally intensive subroutines. In this paper, we reformulate the MML problem as a multi-objective optimization (MOO) problem that overcomes the imbalanced learning issue among modalities and propose a gradient-based algorithm to solve the modified MML problem. We provide convergence guarantees for the proposed method, and empirical evaluations on popular MML benchmarks showcasing the improved performance of the proposed method over existing balanced MML and MOO baselines, with up to ∼ 20 × \sim 20\times reduction in subroutine computation time. Our code is available at https://github.com/heshandevaka/MIMO . Mitigating Modality Imbalance in Multi-modal Learning via Multi-objective Optimization Heshan Fernando 1 Parikshit Ram 2 Yi Zhou 2 Soham Dan 3 \ddagger Horst Samulowitz 2 Nathalie Baracaldo 2 Tianyi Chen 1,4 \dagger 1 Rensselaer Polytechnic Institute 2 IBM Research 3 Microsoft 4 Cornell University 3 3 footnotetext: The work was done when the author was at IBM. 2 2 footnotetext: This work was supported by IBM through the IBM-Rensselaer Future of Computing Research Collaboration. 1 Introduction Multi-modal learning (MML) has gained attention for its ability to leverage information from multiple data modalities, such as images, text, audio, and video ( Wang et al. 2022 ; Shridhar et al. 2020 ; Zhang et al. 2019 ) . An important advancement in this field is the emergence of large multi-modal models (LMMs), which extend MML by scaling model sizes, training on diverse multi-modal datasets, and adopting architectures capable of capturing richer cross-modal interactions. Models such as Gemini ( Team et al. 2023 ) , GPT-4 ( Achiam et al. 2023 ) , and Gato ( Reed et al. 2022 ) have demonstrated exceptional performance across a range of downstream tasks. Unlike uni-modal methods, MML combines complementary information from various sources, leading to richer task representations. For example, combining visual and textual data can enhance understanding in vision-language tasks ( Zhou et al. 2023 ; Oldfield et al. 2023 ; Wei et al. 2024 ) . A common MML approach uses separate encoders for each modality to transform data into feature representations that are “fused” before further processing. These fused feature representations are then processed by a model head to produce the output. In the two-modality case, MML problem can be formulated as: \min_{\vartheta_{mm},\theta_{m_{1}},\theta_{m_{2}}}f_{mm}(\vartheta_{mm},\theta_{m_{1}},\theta_{m_{2}}) (1) where f_{mm} is the multi-modal loss, \theta_{m_{1}} and \theta_{m_{2}} are encoders for modalities m_{1} and m_{2} , and \vartheta_{mm} is the multi-modal head that processes the fused features. The goal is then to optimize f_{mm} in ( 1 ) using standard optimization algorithms (e.g. SGD). Figure 1: Balanced multi-modal learning via multi-objective optimization. (a) : Optimizing the standard MML objective can lead to slower convergence, due to fast to learn modalities dominating the optimization process (b) : We propose MML-via-MOO (MIMO), which optimizes a modified MML objective. This allows the multi-modal network to avoid dominance by one modality, which leads to faster convergence. 1.1 Imbalance issue in multi-modal learning While MML has the potential to outperform uni-modal learning by providing richer task representations, recent studies have shown that standard MML approaches do not always improve performance compared to the best-performing uni-modal models ( Wang et al. 2020 ; Huang et al. 2022 ) . This highlights an inefficiency in current MML methods when it comes to effectively exploiting and integrating information from multiple modalities, presenting a significant challenge to the field. In some recent works ( Ma et al. 2022 ; Hu et al. 2022 ; Peng et al. 2022 ) , this inefficiency is sometimes attributed to the existence of a dominant modality, which prevents the model from fully exploiting the other relatively non-dominant data modalities. Using a toy example, we demonstrate this imbalance in learning different modalities in Figure 1 (a). Note how weights of \theta_{m_{1}} and \theta_{m_{2}} are learned at different speeds (See Figure 1 (a) \|\theta_{m_{1}}\| and \|\theta_{m_{2}}\| curves), leading to slow convergence for the multi-modal model. Recent studies have theoretically explained this inefficiency by investigating the training process of late fusion models. In ( Allen-Zhu and Li 2020 ; Huang et al. 2022 ; Han et al. 2022 ) , the emergence of the dominant modality has been explained via the concept of modality competition. In ( Zhang et al. 2023b ) , the imbalance in modality during learning is attributed to the statistical characteristics of multi-modal datasets. In addition to the theoretical studies, empirical works attempt to develop methods to modulate the training of a multi-modal model and balance the learning of different modalities, and thus, achieve better performance ( Peng et al. 2022 ; Fujimori et al. 2020 ; Yao and Mihalcea 2022 ; Peng et al. 2022 ; Li et al. 2023 ; Wei and Hu 2024 ) (Please refer to Appendix E for an extensive discussion on related works). However, a principled algorithm design that involves low computational complexity per iteration for modality balance in MML with convergence guarantees is missing. 1.2 Our contributions In this work, we aim to address the modality imbalance issue in MML using a modified problem formulation. The key idea behind the modified formulation is to encourage the learning of slow-to-learn modalities. Intuitively, the slow-to-learn modalities have comparatively poor performance, considering the uni-modal objectives. Thus, in the proposed modified problem, we augment the multi-modal objective with modality-specific (uni-modal) objectives, which will be given preference over a multi-modal objective. This preference is enforced by optimizing the multi-modal objective under the constraint of optimality of the worst-performing uni-modal objective. More concretely, we reformulate the MML problem in ( 1 ) as a lexicographic MOO problem ( Miettinen 1999 ) , given by \displaystyle\min_{\vartheta_{mm},\theta_{m_{1}},\theta_{m_{2}}}f_{mm}(\vartheta_{mm},\theta_{m_{1}},\theta_{m_{2}}) (2) \displaystyle\text{s.t}~~\theta_{m_{1}},\theta_{m_{2}},\vartheta_{m_{1}},\vartheta_{m_{2}} \displaystyle~~~~~\in{\arg\!\!\!\!\!\min_{\begin{subarray}{c}\theta_{m_{1}},\theta_{m_{2}}\\ \vartheta_{m_{1}},\vartheta_{m_{2}}\end{subarray}}\max_{k\in\{1,2\}}\left(f_{m_{k}}(\vartheta_{m_{k}},\theta_{m_{k}})-f^{*}_{m_{k}}\right)}. where f_{m_{k}} for k\in\{1,2\} are uni-modal objectives induced by separate uni-modal heads \vartheta_{m_{1}} and \vartheta_{m_{2}} (see Figure 1 (b)), and f^{*}_{m_{k}}=\min_{\bm{\Theta_{m_{k}}},\vartheta_{m_{k}}}f_{m_{k}}(\bm{\Theta_{m_{k}}},\vartheta_{m_{k}}) , for all k\in\{1,2\} . Note that the formulation in ( 2 ) can be easily extended to accommodate an arbitrary number of modalities; however, for the sake of clarity, here we mainly focus on the two-modality case. We then propose MML-via-MOO (MIMO), a simple gradient-based algorithm to solve the new formulation. As shown in Figure 1 (b), our method can alleviate the modality imbalance issue, converging fast to the global optimum. Specifically, note how weights of \theta_{m_{1}} and \theta_{m_{2}} are learned at similar speeds (See Figure 1 (b) \|\theta_{m_{1}}\| and \|\theta_{m_{2}}\| curves), leading to faster convergence. We also establish the convergence of our method theoretically. We then benchmark our algorithm against state-of-the-art MML methods, demonstrating up to a \times 3 reduction in computation time while achieving superior performance. 2 Preliminaries In this section, we introduce some basics in both MML ( Wang et al. 2020 ) and MOO ( Miettinen 1999 ) . 2.1 Multi-modal learning (MML) Consider the classification problem using a multi-modal dataset \mathcal{D}_{mm}:=\{x^{(m_{1})}_{i},x^{(m_{2})}_{i},\dots,x^{(m_{K})}_{i},y_{i}\}_{i=1}^{N} , which consists N input data x^{(m_{k})}_{i} from K modalities m_{k} , where k\!\in\![K]:=\{1,2,\dots,K\} , and the corresponding labels y_{i} .

Read the complete text