See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias

JuneHyoung Kwon 1 * , MiHyeon Kim 1 * † , Eunju Lee 2 , Juhwan Choi 1 , YoungBin Kim 1,2 · 2025

Read See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias by JuneHyoung Kwon 1 * , MiHyeon Kim 1 * † , Eunju Lee 2 , Juhwan Choi 1 , YoungBin Kim 1,2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13834.txt.

Introduction

JuneHyoung Kwon, MiHyeon Kim, Eunju Lee, Juhwan Choi, and YoungBin Kim tackle bias in vision-language models in a 2025 paper. Such models read both images and text to make choices. They often lean too hard on one input type, such as text alone. That lean hurts when the strong input is damaged or weak.

The team shows the flaw comes from uneven training signals, called gradients. Their fix, called BalGrad, reweights signals from each input type by its share of the work. It also lines up signals from distinct tasks so they do not clash. Tests on three datasets, from food photos to memes, show less lean on one input. The model stays strong even when one input is impaired.

Catalog record

CREATORJuneHyoung Kwon 1 * , MiHyeon Kim 1 * † , Eunju Lee 2 , Juhwan Choi 1 , YoungBin Kim 1,2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13834

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias JuneHyoung Kwon 1 * , MiHyeon Kim 1 * † , Eunju Lee 2 , Juhwan Choi 1 , YoungBin Kim 1,2 Abstract Vision-language (VL) models have demonstrated strong performance across various tasks. However, these models often rely on a specific modality for predictions, leading to “dominant modality bias.” This bias significantly hurts performance, especially when one modality is impaired. In this study, we analyze model behavior under dominant modality bias and theoretically show that unaligned gradients or differences in gradient magnitudes prevent balanced convergence of the loss. Based on these findings, we propose a novel framework, BalGrad to mitigate dominant modality bias. Our approach includes inter-modality gradient reweighting, adjusting the gradient of KL divergence based on each modality’s contribution, and inter-task gradient projection to align task directions in a non-conflicting manner. Experiments on UPMC Food-101, Hateful Memes, and MM-IMDb datasets confirm that BalGrad effectively alleviates over-reliance on specific modalities when making predictions. † † footnotetext: * Equal contribution. † † footnotetext: † Currently at: KT CORPORATION, [email protected] . 1 Introduction Vision-language (VL) models combine image and text modalities, resulting in powerful multi-modal representations. Owing to this integration of two modalities, these models can achieve higher performance in vision-language tasks. Recently, leveraging extensive datasets, VL models have demonstrated remarkable performance across various tasks such as image captioning Hu et al. 2022 , visual question answering Khademi et al. 2023 , and cross-modal retrieval Liu et al. 2023 , showcasing their capability to harness the complementary strengths of visual and textual data. However, these models often rely on a single modality rather than treating and utilizing them equally, leading to the dominance of a certain modality on the overall performance. A conceptual overview of this effect can be seen in Figure 1 . This phenomenon, where a specific modality disproportionately influences the model’s outcomes, is referred to as “dominant modality bias” Woo et al. 2023 . For instance, VL models tend to be biased towards the text modality when recognizing hate expressions Kiela et al. 2020 ; Aggarwal et al. 2024 , thereby limiting the VL model’s ability to effectively integrate and interpret images. Figure 1: Conceptual visualization of dominant modality bias. The key modality differs by task: (a) For the hate recognition task, text descriptions of memes lead, while (b) for the food classification task, food images play a crucial role in prediction. This bias behaves particularly detrimentally when one modality is impaired, such as when data is noisy and it is difficult to gather paired data Garg et al. 2022 ; Woo et al. 2023 ; Yang et al. 2024 . This issue is common in real-world scenarios due to privacy-related data sharing restrictions or stringent data storage policies Voigt and Von dem Bussche 2017 and can severely degrade the model’s performance. Additionally, the failure to sufficiently explore the weak modality limits the overall performance of the VL model Wang et al. 2020 ; Huang et al. 2022 ; Peng et al. 2022 , highlighting the need for robust solutions to mitigate dominant modality bias. To address this issue and balance the information between modalities, numerous studies have been conducted. Several studies have focused on modulating the gradients of each encoder based on the confidence of individual modalities Peng et al. 2022 ; Li et al. 2023 . Other approaches have involved training multimodal models using the best-performing learning rates from unimodal models Yao and Mihalcea 2022 . However, these methods often induce negative transfer Wang et al. 2019 ; Yu et al. 2020 , which occurs when the model’s performance decreases with the addition of modality data compared to solely using unimodal data. We first analyze the behavior of models after the dominant modality bias has taken root. Our analysis reveals that certain modalities are more crucial to target performance and observes that the dominant and weak modalities converge at different rates during training. Additionally, we theoretically demonstrate that the balanced convergence of the loss is influenced by both the magnitude and direction of the gradient. Based on these findings, we propose BalGrad (Balancing Gradients) to mitigate dominant modality bias. Firstly, we adopt a mutual KL divergence between the predictions of each modality to ensure balanced updates. However, a naive approach that equally aligns the distributions of two modalities can hinder the representation learning of each modality. To address this, we introduce inter-modality gradient reweighting , which adjusts the magnitude of the gradient of the KL divergence term based on the learning status of each modality. Additionally, we propose inter-task gradient projection , which updates the gradient of the target task to establish a balance between both modalities. We project the target task’s gradient in a direction orthogonal to the KL divergence gradient if a conflict between the gradients occurs, encouraging stabilized training between the two modalities. We evaluate the effectiveness of BalGrad on models using three vision-language datasets: UPMC Food-101 Wang et al. 2015 , Hateful Memes Kiela et al. 2020 , and MM-IMDb Arevalo et al. 2017 . To simulate the influence of individual modalities, we conduct experiments under conditions where specific modalities are missing or impaired by noise. The experimental results demonstrate that the proposed method reduces the gap between the modalities while avoiding negative transfer. The contributions of our proposed method are as follows: • We analyze the dominant modality bias and theoretically demonstrate that the balanced convergence of loss is influenced by both the magnitude and direction of the gradient. • We propose BalGrad , which reweights the gradients between modalities to ensure stable convergence and projects the target task’s gradient to avoid conflicts that hinder balanced learning. • Experimental results across UPMC Food-101, Hateful Memes, and MM-IMDb under different impaired conditions confirm the effectiveness of our proposed method in mitigating dominant modality bias. 2 Related Work Figure 2: Experimental results on the UPMC Food-101, Hateful Memes, and MM-IMDb datasets in the presence of dominant modality bias. (a) Performance visualization under different missing conditions (full, image only (missing text), text only (missing image)) for each dataset. (b) Illustration of learning curves for each modality across datasets. In multimodal models, such as VL models, a bias towards a preferred or easier-to-learn modality often leads to the under-exploration of others Wang et al. 2020 ; Huang et al. 2022 ; Peng et al. 2022 . Studies have analyzed this, noting that multimodal models are prone to overfitting and show discrepancies in generalization across modalities Wang et al. 2020 . Differences in convergence speeds also contribute to this bias Yao and Mihalcea 2022 ; Wu et al. 2022 . An early study in this field finds that certain modalities, correlating with their network’s random initialization, dominate the learning process Huang et al. 2022 , while other researchers attribute the preference to unimodal representation margins and insufficient integration of modalities Yang et al. 2024 . Another line of study highlights that spurious correlations with instance labels cause imbalances in modality utilization Guo et al. 2023 . In this paper, we identify that the dominant modality bias in VL models arises from the influence of gradient magnitude and direction on the model’s loss

Read the complete text