Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking
Reza Khanmohammadi, Erfan Miahi, Simerjot Kaur, Charese H. Smiley, Ivan Brugere, Kundan Thind, Mohammad M. Ghassemi · 2026
Read Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking by Reza Khanmohammadi, Erfan Miahi, Simerjot Kaur, Charese H. Smiley, Ivan Brugere, Kundan Thind, Mohammad M. Ghassemi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10893.txt.
Description
Reza Khanmohammadi and six colleagues tackle a quiet failure of vision-language models: fluent, confident answers driven by language habits alone, with the image adding nothing. Standard confidence scores cannot catch this, since they watch only normal inference and never test whether the picture mattered.
Their BICR framework trains a light probe on hidden states from a frozen model, shown twice: once with the real image and once with it blacked out. A ranking loss punishes confidence that stays high when the picture vanishes, teaching the probe to treat visual grounding as a reliability signal at no extra inference cost. Across five models and tasks from medical imaging to finance documents, BICR leads on calibration and discrimination together.
Catalog record
| CREATOR | Reza Khanmohammadi, Erfan Miahi, Simerjot Kaur, Charese H. Smiley, Ivan Brugere, Kundan Thind, Mohammad M. Ghassemi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 263046 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10893