Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
Chufan Shi, Cheng Yang, Yaokang Wu, Linghao Jin, Bo Shui, Taylor Berg-Kirkpatrick, Xuezhe Ma · 2026
Read Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination by Chufan Shi, Cheng Yang, Yaokang Wu, Linghao Jin, Bo Shui, Taylor Berg-Kirkpatrick, Xuezhe Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15864.txt.
Description
Chufan Shi and six colleagues ask whether vision-language models that say 'let me check the figure again' look again. Their VisualSwap probe swaps the image for a different one after reasoning, then tests whether the model notices. On VS-Bench, 800 pairs from MathVista, MathVerse, MathVision and MMMU-Pro, Qwen3-VL, Kimi-VL and ERNIE-VL miss the swap, with accuracy falling by up to 60 percent. Thinking models prove three times more vulnerable than instructed ones, and scaling offers no fix. User prompts restore grounding, but own reflections do not.
Catalog record
| CREATOR | Chufan Shi, Cheng Yang, Yaokang Wu, Linghao Jin, Bo Shui, Taylor Berg-Kirkpatrick, Xuezhe Ma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73230 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15864