Disparities in Negation Understanding Across Languages in Vision-Language Models

Charikleia Moraitaki, Sarah Pan, Skyler Pulling, Gwendolyn FluscheKumail AlhamoudMarzyeh Ghassemi Massachusetts Institute of Technology · 2026

Read Disparities in Negation Understanding Across Languages in Vision-Language Models by Charikleia Moraitaki, Sarah Pan, Skyler Pulling, Gwendolyn FluscheKumail AlhamoudMarzyeh Ghassemi Massachusetts Institute of Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18942.txt.

Introduction

Moraitaki and five colleagues introduce the first human checked multilingual negation benchmark for VLMs. It spans seven languages: English, Mandarin Chinese, Arabic, Greek, Russian, Tagalog, and Spanish. Such models show affirmation bias, tending to select positive captions even when the correct description contains negation. Standard CLIP scores no better than chance on non-Latin scripts, while MultiCLIP has the highest and most uniform accuracy. SpaceVLM, a new negation correction, lifts several languages greatly but with varied results.

Catalog record

CREATORCharikleia Moraitaki, Sarah Pan, Skyler Pulling, Gwendolyn FluscheKumail AlhamoudMarzyeh Ghassemi Massachusetts Institute of Technology
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE11340
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18942

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Disparities in Negation Understanding Across Languages in Vision-Language Models Charikleia Moraitaki, Sarah Pan, Skyler Pulling, Gwendolyn FluscheKumail AlhamoudMarzyeh Ghassemi Massachusetts Institute of Technology Abstract Vision-language models (VLMs) exhibit affirmation bias : a systematic tendency to select positive captions (“X is present”) even when the correct description contains negation (“no X”). While prior work has documented this failure mode in English and proposed solutions, negation manifests differently across languages through varying morphology, word order, and cliticization patterns–raising the question of whether these solutions serve all linguistic communities equitably. We introduce the first human-verified multilingual negation benchmark, spanning seven typologically diverse languages: English, Mandarin Chinese, Arabic, Greek, Russian, Tagalog, and Spanish. Evaluating three VLMs–CLIP, SigLIP, and MultiCLIP–we find that standard CLIP performs at or below chance on non-Latin-script languages, while MultiCLIP achieves the highest and most uniform accuracy. We also evaluate SpaceVLM, a proposed negation correction, and find that it produces substantial improvements for several languages–particularly English, Greek, Spanish, and Tagalog–while showing varied effectiveness across typologically different languages. This variation reveals that linguistic properties like morphology, script, and negation structure interact with model improvements in fairness-relevant ways. As VLMs are deployed globally, multilingual benchmarks are essential for understanding not just whether solutions work, but for whom. 1 Introduction Prompts to language models are overwhelmingly given in the affirmative: “generate an image with a dog,” “give me a list that contains recipes.” What happens when we negate? Recent work by 1 demonstrates that state-of-the-art vision-language models (VLMs) frequently fail on negated queries, exhibiting affirmation bias : the tendency to match a caption like “there is no boat” to an image containing a boat, simply because the word “boat” appears. This failure has serious downstream consequences—in radiology, the distinction between “no pleural effusion” and “pleural effusion present” determines whether a finding is flagged or dismissed. However, all existing evaluations of negation understanding focus exclusively on English. This is a significant gap because negation is not always expressed the same way. For example, English uses simple adverbial particles (“no,” “not”), but Greek employs verbal negation with existential constructions (“ \delta\varepsilon\nu \upsilon\pi\acute{\alpha}\rho\chi\varepsilon\iota ” = “there does not exist”), Arabic uses cliticized markers in a right-to-left script, and Chinese relies on isolating particles with distinct semantic functions. These structural differences may influence VLM behavior in ways that English-only benchmarks cannot reflect. This is crucial for fairness, because LLMs are disproportionately built around English and a small handful of other high-resource languages, creating a “digital language divide” that systematically disadvantages most of the world’s languages ( 2 ) . Prior work has documented systematic performance inequalities across languages in NLP more broadly ( 3 ) , and if VLMs similarly handle negation better for some linguistic communities than others, this constitutes an inequity that deserves attention—particularly as these systems enter safety-critical domains. Our primary contribution is the first multilingual negation benchmark for VLMs, spanning seven typologically diverse languages. Using this benchmark, we find that even explicitly multilingual VLMs exhibit significant cross-lingual negation gaps, with performance varying by up to 27.5 percentage points across languages. We further apply SpaceVLM ( 8 ) —a recently proposed negation correction—across all languages and uncover that the method’s effectiveness correlates with how a language expresses negation morphologically. This variation is itself informative: it reveals that linguistic structure shapes model behavior in ways that inform equitable deployment. 2 Approach Benchmark construction. We extend the English NegBench dataset ( 1 ) , which contains 5,914 image-caption sets from COCO ( 5 ) with 4-way multiple choice, to seven languages chosen for typological diversity: English, Spanish, Greek, and Tagalog (negation via independent particles), Russian and Arabic (morphologically complex negation), and Mandarin Chinese (isolating, with distinct negation particles for different semantic functions). These languages also span Latin and non-Latin scripts, left-to-right and right-to-left writing, and multiple language families. Translations used Google Translate followed by human verification from native speakers ( 30 samples/language), who checked that negation markers were correctly translated, that negated sentences remained semantically faithful to the English source, and that phrasing was natural in the target language. Models and evaluation. We evaluate three contrastive VLMs representing different training regimes: CLIP ( 7 ) (primarily English-trained), SigLIP ( 10 ) (multilingual), and MultiCLIP ( 4 ) (multilingual). We also evaluate after applying SpaceVLM ( 8 ) , which computes hybrid embeddings that decompose captions into affirmative and negated components using a threshold \tau{=}0.92 derived from English data. We measure top-1 accuracy on the 4-way caption ranking task (chance = 25%). 3 Results Even multilingual VLMs exhibit significant negation gaps. Table 1 summarizes baseline performance. CLIP shows the most extreme disparity: English achieves 39.3% while Arabic (15.7%), Tagalog (11.8%), and Greek (18.0%) fall below chance, indicating systematic failure. SigLIP, trained on multilingual data, narrows this gap but still shows a 7.4-point spread. MultiCLIP achieves the most consistent performance (std. dev. 1.3%), yet its mean accuracy of 41.2% indicates that even the best multilingual model struggles with negation across the board. Table 1: Baseline cross-lingual negation performance. Even MultiCLIP, the most equitable model, achieves only 41% mean accuracy, showing that multilingual training alone does not resolve negation understanding. Model Mean Acc. Std. Dev. Max–Min Gap CLIP 23.5% 9.2% 27.5% SigLIP 30.5% 2.6% 7.4% MultiCLIP 41.2% 1.3% 4.4% SpaceVLM as a diagnostic lens. SpaceVLM produces substantial improvements for several languages, but the gains are not uniform and follow a typological pattern (Table 2 ). Languages with negation expressed via independent particles ( 6 ) , such as English, Spanish, Greek, and Tagalog, show consistent large gains across all three models (+9.0 to +27.5pp). By contrast, languages with morphologically complex negation ( 6 ) (Russian, Arabic) or distinct negation particles with different semantic functions ( 6 ) (Chinese) show smaller or variable effects. Table 2: SpaceVLM reveals typological patterns ( \Delta = SpaceVLM - Baseline). Green : gain > 5pp; Red : decrease > 5pp. Languages with adverbial negation benefit consistently; languages with complex morphological negation show variable effects, revealing that negation structure affects behavior. CLIP SigLIP MultiCLIP Language Base Space \Delta Base Space \Delta Base Space \Delta English 39.3 62.5 +23.3 34.6 54.6 +20.0 40.6 66.1 +25.5 Chinese 25.5 30.7 +5.2 31.9 12.4 - 19.5 42.5 38.6 - 3.9 Arabic 15.7 16.6 +0.9 28.3 30.1 +1.8 40.6 34.5 - 6.1 Greek 18.0 33.9 +15.9 28.1 46.0 +17.9 38.9 66.4 +27.5 Russian 20.4 19.1 - 1.3 30.4 19.8 - 10.6 43.3 36.3 - 7.0 Tagalog 11.8 50.7 +38.9 27.2 31.0 +3.8 41.6 50.6 +9.0 Spanish 33.6 56.2 +22.6 33.1 56.8 +23.7 41.3 67.5 +26.2 The pattern is robust. We applied SpaceVLM to NegCLIP and ConCLIP—models fine-tuned specifically for negation on English data ( 9 ) and it produces the same typological pattern: strong English gains (57–69%) with consistent gaps for Arabic (18–20%), Russian (27–29%), and Chinese (35–39%). This persistence across five model-solution combinations suggests it reflects a genuine property of how these languages encode negation. 4 Discussion: Implications for Equitable Deployment VLMs handle negation better in English than in other languages, which constitutes a clear disparity in model reliability across linguistic communities. This is especially concerning in safety-critical applications where differing results provide unequal service quality along linguistic lines. Our benchmark makes these disparities measurable. The typological pattern we observe

Read the complete text