MEBench: A Novel Benchmark for Understanding Mutual Exclusivity Bias in Vision-Language Models

Anh Thai, Stefan Stojanov, Zixuan Huang, Bikram Boote, James M. Rehg · 2025

Read MEBench: A Novel Benchmark for Understanding Mutual Exclusivity Bias in Vision-Language Models by Anh Thai, Stefan Stojanov, Zixuan Huang, Bikram Boote, James M. Rehg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20122.txt.

Description

Thai and co-authors present a 2025 benchmark on a word-learning trick in vision-language models. Young children use mutual exclusivity: a new word likely names the object they do not know. Models face the same choice in apps, yet the bias was untested. Their bench, MEBench, adds spatial reasoning to make scenes harder and more real.

A flexible pipeline builds varied annotated scenes for controlled tests. Fresh scoring rules capture how models use the bias. The tested models showed only weak bias, the team found. Extra spatial clues did help them settle some hard cases. The bench and pipeline are public for other labs to extend.

Catalog record

CREATORAnh Thai, Stefan Stojanov, Zixuan Huang, Bikram Boote, James M. Rehg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20122

Browse the full library · About the Tower of Babel