CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models
Reem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid, Zaid Alyafeai, Abdelrahman Eldesokey, Bernard Ghanem · 2026
Read CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models by Reem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid, Zaid Alyafeai, Abdelrahman Eldesokey, Bernard Ghanem (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17826.txt.
Description
Reem Alzahrani and six colleagues ask whether vision-language models count what they see or what they know. Their CounterCount dataset pairs factual images with counterfactual ones, where an attribute count is atypical. Humans score perfectly on both types, but across 16 models, accuracy falls from 91.6% factual to 49.5% counterfactual. Most errors match the canonical count, revealing a bias toward knowledge priors over visual evidence. Models behave System-1-like, recognizing the subject shallowly and underusing visual evidence, though cueing and attention boosts can reduce the bias.
Catalog record
| CREATOR | Reem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid, Zaid Alyafeai, Abdelrahman Eldesokey, Bernard Ghanem |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 113888 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17826