Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination

Chufan Shi, Cheng Yang, Yaokang Wu, Linghao Jin, Bo Shui, Taylor Berg-Kirkpatrick, Xuezhe Ma · 2026

Read Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination by Chufan Shi, Cheng Yang, Yaokang Wu, Linghao Jin, Bo Shui, Taylor Berg-Kirkpatrick, Xuezhe Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15864.txt.

Description

Chufan Shi and six colleagues ask whether vision-language models that say 'let me check the figure again' look again. Their VisualSwap probe swaps the image for a different one after reasoning, then tests whether the model notices. On VS-Bench, 800 pairs from MathVista, MathVerse, MathVision and MMMU-Pro, Qwen3-VL, Kimi-VL and ERNIE-VL miss the swap, with accuracy falling by up to 60 percent. Thinking models prove three times more vulnerable than instructed ones, and scaling offers no fix. User prompts restore grounding, but own reflections do not.

Catalog record

CREATORChufan Shi, Cheng Yang, Yaokang Wu, Linghao Jin, Bo Shui, Taylor Berg-Kirkpatrick, Xuezhe Ma
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73230
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15864

Browse the full library · About the Tower of Babel