MASS: Overcoming Language Bias in Image-Text Matching

Jiwan Chung, Seungwon Lim, Sangkyu Lee, Youngjae Yu · 2025

Read MASS: Overcoming Language Bias in Image-Text Matching by Jiwan Chung, Seungwon Lim, Sangkyu Lee, Youngjae Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11469.txt.

Introduction

Jiwan Chung, Seungwon Lim, Sangkyu Lee, and Youngjae Yu published this paper in 2025. It fixes a flaw in AI models that match pictures with captions. Such models often lean on language habits instead of truly looking at the image, so they pick captions that sound right rather than captions that fit.

Their method, called MASS, scores how strongly a picture and a text are truly tied to each other, beyond what words alone suggest. It plugs into existing models with no extra training. In tests, the method cut the language bias while keeping the model's grasp of word order and detail. Better matching matters for search, captioning, and any tool that pairs images with text. The authors present MASS as a light add-on that builders can adopt at once.

Catalog record

CREATORJiwan Chung, Seungwon Lim, Sangkyu Lee, Youngjae Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-11469

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MASS: Overcoming Language Bias in Image-Text Matching Jiwan Chung, Seungwon Lim, Sangkyu Lee, Youngjae Yu Abstract Pretrained visual-language models have made significant advancements in multimodal tasks, including image-text retrieval. However, a major challenge in image-text matching lies in language bias, where models predominantly rely on language priors and neglect to adequately consider the visual content. We thus present Multimodal ASsociation Score (MASS), a framework that reduces the reliance on language priors for better visual accuracy in image-text matching problems. It can be seamlessly incorporated into existing visual-language models without necessitating additional training. Our experiments have shown that MASS effectively lessens language bias without losing an understanding of linguistic compositionality. Overall, MASS offers a promising solution for enhancing image-text matching performance in visual-language models. 1 Yonsei University 50 Yonsei-ro, Seodaemun-gu Seoul, South Korea [email protected] 1 Introduction Recently, pretrained visual-language models have showcased impressive performance across a broad spectrum of multimodal tasks ( Li et al. 2022 ; Alayrac et al. 2022 ; Wang et al. 2022b ) , including simple image-text retrieval ( Chen et al. 2015 ; Young et al. 2014 ) . In particular, CLIP ( Radford et al. 2021 ) has emerged as one of the most popular models for image-text matching. The image-text similarity score derived from CLIP is shown to be effective for assessing image caption quality ( Hessel et al. 2021 ) , providing reward signals to train multimodal models ( Yu et al. 2023 ) , and building a large-scale image search system ( Beaumont 2022 ) . However, CLIP falters when the given task requires accurate language modeling. Recent benchmarks have unveiled CLIP’s deficiency in modeling linguistic compositionality ( Thrush et al. 2022 ; Nikolaus et al. 2022 ; Yuksekgonul et al. 2022 ) . Furthermore, current visual-language models also have demonstrated inconsistent outcomes on image-text matching that demand comprehension of linguistic constructs, such as the existence and quantity of objects or coreference ( Parcalabescu et al. 2022 ; Shekhar et al. 2017 ) . Figure 1: Captions retrieved with each method given the image, where only MASS succeeds in ruling out the failure modes. Models trained Image-Text Contrastive (ITC) objectives such as CLIP ( Radford et al. 2021 ) fail to model linguistic structure. Token Likelihood (TL) of image captioning models including OFA ( Wang et al. 2022b ) shows overreliance on its language prior. Our MASS amends the language bias of image captioning models for accurate image-text matching capability. Figure 2: (a) Given an image of a girl playing tennis, the visual-language model falsely retrieves captions describing the subject as male by relying on language bias. (b) On the other hand, MASS reduces such gender bias by adopting pointwise mutual information which suppresses the text-only marginal likelihood. We provide the corresponding experimental results in section 4.3 . As an alternative to the contrastive objective in CLIP, recent studies ( Petryk et al. 2024 ; Tschannen et al. 2024 ) revealed that log-likelihood induced from the autoregressive image captioning objective features a better understanding of linguistic structures, including compositionality. The objective fosters language generation capability in the trained models, inducing stronger linguistic understanding. Still, there is a caveat in directly using the image captioning models to assess image-text similarity. While training for the captioning objective, the models inherently build a prior for language distribution on what text sequence is more likely regardless of the given image . As shown in Figure 1 , overreliance on this prior leads to weaker visual conditioning and then to incorrect image-text matching. Drawing inspiration from previous research ( Niu et al. 2020 ) , we refer to such phenomenon as language bias . Language bias refers to the propensity of visual-language models to rely heavily on language priors in the training data instead of properly conditioning their output on the given images. It has been repeatedly reported that language bias is a major bottleneck in visual question-answering ( Agrawal, Batra, and Parikh 2016 ; Zhang et al. 2016 ; Agrawal et al. 2018 ; Si et al. 2022 ) , and other vision-language tasks ( Srinivasan and Bisk 2021 ; Salin et al. 2022 ) . In this paper, we present Multimodal ASsociation Score (MASS) as an inference-time framework designed to reduce language bias in image-text matching. MASS aims to measure the strength of the association between image and text modality without involving the textual prior. Our approach uniquely functions as follows: Given a pretrained visual-language model, we first extract the image-conditional and text-only likelihood per text token. We then compute the pointwise mutual information of each image and text token and aggregate them for the text sequence to create a debiased scalar similarity score. Importantly, MASS can be computed for any off-the-shelf visual-language model that outputs image-conditioned text likelihood. Also, it does not require any additional training. We first demonstrate that MASS can mitigate language bias in visual-language models by evaluating it with multimodal color ( section 4.1 ), number ( section 4.2 ), and gender ( section 4.3 ) bias benchmarks. Our results indicate that MASS significantly improves performance over CLIP ( Radford et al. 2021 ) and raw token likelihood (TL) in the datasets with clear language bias (color), explicit bias in labels (number), and practical image search problems (gender) about reliance on language bias. We further evaluate MASS’s capability to maintain linguistic understanding against two visual-linguistic compositionality benchmarks; Winoground ( Thrush et al. 2022 ) and SVO-Probe ( Hendricks and Nematzadeh 2021 ) . MASS greatly enhances the performance of the backbone visual-language model in both experiments and outperforms the strong baselines. The main contributions of this paper are threefold. Firstly, we present MASS as an effective framework for language bias reduction in image-text matching tasks. Secondly, we demonstrate that MASS can both reduce language bias and maintain linguistic understanding with various biases and compositionality benchmarks. Lastly, our approach does not necessitate additional training and can be directly applied to a wide variety of visual-language models for robust image-text matching. 2 Image-Text Similarity Functions Our objective is to identify a similarity function that accurately pairs image and text, even in the presence of language bias in the pretrained model. Given an image \mathbf{c}\in\mathcal{C} and a text \mathbf{x}\in\mathcal{X} , the similarity function \mathcal{S}(\mathbf{c},\mathbf{x}) evaluates the closeness of alignment such that parallel image-text pairs have high scores. We divide the similarity functions by the granularity of the training objectives: Image-Text Contrastive Learning (ITC) and Image-Text Matching (ITM) provide scores on sequence-level ( section 2.1 ), while Token-Level supervision (TL) such as supervised image captioning applies token-level ( section 2.2 ) feedback. 2.1 Sequence-Level Similarity Functions The sequence-level similarity functions are defined on top of dual vision-language encoders. The dual encoders may be either separate ( i.e., architectures with independent encoders per modality as CLIP ( Radford et al. 2021 ) and the lower modules of ALBEF ( Li et al. 2021 ) ) or intertwined ( e.g., the upper module of ALBEF) in terms of architecture. Image-Text Contrastive Learning . ITC is a training objective for learning multimodal joint embedding space based on the Noise Contrastive Estimation ( Oord, Li, and Vinyals 2018 ) . We use separate image and text encoders f_{\phi} ,

Read the complete text