Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?

Ta Duc Huy, Trang Nguyen, Townim Chowdhury, Ankit Yadav, Minh-Son To, Zhibin Liao, Johan W.Verjans, Vu Minh Hieu Phan · 2026

Read Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity? by Ta Duc Huy, Trang Nguyen, Townim Chowdhury, Ankit Yadav, Minh-Son To, Zhibin Liao, Johan W.Verjans, Vu Minh Hieu Phan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31407.txt.

Description

Ta and seven colleagues propose Visual Semantic Entropy, a way to tell when a vision-language model should doubt its own answer. Models often answer ambiguous images with false confidence. Standard semantic entropy reads output diversity, but overconfident visual embeddings suppress that diversity, so doubt gets understated.

Perturbation methods that rewrite the text mostly measure prompt sensitivity instead. Visual Semantic Entropy perturbs only the image and holds the question fixed. It clusters the answers into semantic prototypes and scores their weighted spread. Across five visual question answering benchmarks and five modern vision-language models, the method captures visual ambiguity best and sets a new state of the art for uncertainty estimation. The code is public for reuse.

Catalog record

CREATORTa Duc Huy, Trang Nguyen, Townim Chowdhury, Ankit Yadav, Minh-Son To, Zhibin Liao, Johan W.Verjans, Vu Minh Hieu Phan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31407

Browse the full library · About the Tower of Babel