Know What You do Not Know: Verbalized Uncertainty Estimation Robustness on Corrupted Images in Vision-Language Models

Mirko Borszukovszki, Ivo Pascal de Jong, Matias Valdenegro-Toro · 2025

Read Know What You do Not Know: Verbalized Uncertainty Estimation Robustness on Corrupted Images in Vision-Language Models by Mirko Borszukovszki, Ivo Pascal de Jong, Matias Valdenegro-Toro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03440.txt.

Description

Mirko Borszukovszki, Ivo Pascal de Jong, and Matias Valdenegro-Toro tested whether vision-language models know when they are unsure in 2025. A model that can state its own uncertainty helps users trust or check its answers. Most past work on this skill used text-only models.

The team ran three leading vision-language models on images damaged to varied degrees. As damage grew worse, the models grew worse at judging their own certainty. In most tests the models were also overconfident, claiming more certainty than their accuracy deserved. The results show that stated uncertainty in these models is fragile when image quality drops, a risk for real-world use.

Catalog record

CREATORMirko Borszukovszki, Ivo Pascal de Jong, Matias Valdenegro-Toro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-03440

Browse the full library · About the Tower of Babel