Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?
Ta Duc Huy, Trang Nguyen, Townim Chowdhury, Ankit Yadav, Minh-Son To, Zhibin Liao, Johan W.Verjans, Vu Minh Hieu Phan · 2026
Read Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity? by Ta Duc Huy, Trang Nguyen, Townim Chowdhury, Ankit Yadav, Minh-Son To, Zhibin Liao, Johan W.Verjans, Vu Minh Hieu Phan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31407.txt.
Description
Ta and seven colleagues propose Visual Semantic Entropy, a way to tell when a vision-language model should doubt its own answer. Models often answer ambiguous images with false confidence. Standard semantic entropy reads output diversity, but overconfident visual embeddings suppress that diversity, so doubt gets understated.
Perturbation methods that rewrite the text mostly measure prompt sensitivity instead. Visual Semantic Entropy perturbs only the image and holds the question fixed. It clusters the answers into semantic prototypes and scores their weighted spread. Across five visual question answering benchmarks and five modern vision-language models, the method captures visual ambiguity best and sets a new state of the art for uncertainty estimation. The code is public for reuse.
Catalog record
| CREATOR | Ta Duc Huy, Trang Nguyen, Townim Chowdhury, Ankit Yadav, Minh-Son To, Zhibin Liao, Johan W.Verjans, Vu Minh Hieu Phan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.31407