Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?

Zixuan Lan University of Chicago [email protected] Luzhe Sun Toyota Technological Institute at Chicago [email protected] Matthew R. WalterToyota Technological Institute at Chicago [email protected] Jiawei Zhou Stony Brook University [email protected] · 2026

Read Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision? by Zixuan Lan University of Chicago [email protected] Luzhe Sun Toyota Technological Institute at Chicago [email protected] Matthew R. WalterToyota Technological Institute at Chicago [email protected] Jiawei Zhou Stony Brook University [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22903.txt.

Description

Lan and three colleagues test whether vision-language benchmarks truly test vision, after finding that deleting many image tokens barely dented scores on a hallucination benchmark. Across open-source models and several levels of degradation, occlusion, and question reformulation, predictions proved far less sensitive to fine visual evidence than accuracy implied. Even unchanged predictions hid weakened internal support for the right answer. Deeper layers showed visual tokens growing more similar, a possible explanation.

Catalog record

CREATORZixuan Lan University of Chicago [email protected] Luzhe Sun Toyota Technological Institute at Chicago [email protected] Matthew R. WalterToyota Technological Institute at Chicago [email protected] Jiawei Zhou Stony Brook University [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62553
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22903

Browse the full library · About the Tower of Babel