Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference

Zhuo Chen, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinyu Geng, Pengjun Xie, Fei Huang, Kewei Tu · 2025

Read Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference by Zhuo Chen, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinyu Geng, Pengjun Xie, Fei Huang, Kewei Tu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18023.txt.

Description

Zhuo Chen and co-authors teach vision-language models in 2025 to know what they do not know. These models falter on questions needing fresh or deep facts. Fetching outside help works but costs too much when done for every question. Their method fine-tunes a model on an automatically built set to flag when a question sits beyond its knowledge.

Tests on several visual question sets showed the boundary drawn well. Using it, the team cut needless fetching while keeping or raising accuracy. One model's boundary even served as a stand-in for others. Smarter fetching decisions could make such systems cheaper and faster in daily use.

Catalog record

CREATORZhuo Chen, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinyu Geng, Pengjun Xie, Fei Huang, Kewei Tu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18023

Browse the full library · About the Tower of Babel