Probing Visual Language Priors in VLMs

Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, Honglak Lee · 2025

Read Probing Visual Language Priors in VLMs by Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, Honglak Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00569.txt.

Description

Tiange Luo and four co-authors test in 2024 whether vision-language models truly look at images or lean on text habits learned in training. Such models answer questions about pictures, yet training data can plant strong guesses. A model may answer from habit even when the image says otherwise.

Their test set, ViLP, uses made-up images that break normal patterns. Each question offers three answers and three images: one fits text habit alone, and two demand real looking. People score near perfect, while a top model scores only about two-thirds right. A self-training fix, where models build new data and train on clean versus spoiled image pairs, lifts scores for open models.

Catalog record

CREATORTiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, Honglak Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-00569

Browse the full library · About the Tower of Babel