Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
Zixuan Lan University of Chicago [email protected] Luzhe Sun Toyota Technological Institute at Chicago [email protected] Matthew R. WalterToyota Technological Institute at Chicago [email protected] Jiawei Zhou Stony Brook University [email protected] · 2026
Read Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision? by Zixuan Lan University of Chicago [email protected] Luzhe Sun Toyota Technological Institute at Chicago [email protected] Matthew R. WalterToyota Technological Institute at Chicago [email protected] Jiawei Zhou Stony Brook University [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22903.txt.
Description
Lan and three colleagues test whether vision-language benchmarks truly test vision, after finding that deleting many image tokens barely dented scores on a hallucination benchmark. Across open-source models and several levels of degradation, occlusion, and question reformulation, predictions proved far less sensitive to fine visual evidence than accuracy implied. Even unchanged predictions hid weakened internal support for the right answer. Deeper layers showed visual tokens growing more similar, a possible explanation.
Catalog record
| CREATOR | Zixuan Lan University of Chicago [email protected] Luzhe Sun Toyota Technological Institute at Chicago [email protected] Matthew R. WalterToyota Technological Institute at Chicago [email protected] Jiawei Zhou Stony Brook University [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62553 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22903