CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models

Reem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid, Zaid Alyafeai, Abdelrahman Eldesokey, Bernard Ghanem · 2026

Read CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models by Reem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid, Zaid Alyafeai, Abdelrahman Eldesokey, Bernard Ghanem (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17826.txt.

Description

Reem Alzahrani and six colleagues ask whether vision-language models count what they see or what they know. Their CounterCount dataset pairs factual images with counterfactual ones, where an attribute count is atypical. Humans score perfectly on both types, but across 16 models, accuracy falls from 91.6% factual to 49.5% counterfactual. Most errors match the canonical count, revealing a bias toward knowledge priors over visual evidence. Models behave System-1-like, recognizing the subject shallowly and underusing visual evidence, though cueing and attention boosts can reduce the bias.

Catalog record

CREATORReem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid, Zaid Alyafeai, Abdelrahman Eldesokey, Bernard Ghanem
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE113888
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17826

Browse the full library · About the Tower of Babel