Probing Visual Language Priors in VLMs
Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, Honglak Lee · 2025
Read Probing Visual Language Priors in VLMs by Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, Honglak Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00569.txt.
Description
Tiange Luo and four co-authors test in 2024 whether vision-language models truly look at images or lean on text habits learned in training. Such models answer questions about pictures, yet training data can plant strong guesses. A model may answer from habit even when the image says otherwise.
Their test set, ViLP, uses made-up images that break normal patterns. Each question offers three answers and three images: one fits text habit alone, and two demand real looking. People score near perfect, while a top model scores only about two-thirds right. A self-training fix, where models build new data and train on clean versus spoiled image pairs, lifts scores for open models.
Catalog record
| CREATOR | Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, Honglak Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-00569