Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference
Zhuo Chen, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinyu Geng, Pengjun Xie, Fei Huang, Kewei Tu · 2025
Read Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference by Zhuo Chen, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinyu Geng, Pengjun Xie, Fei Huang, Kewei Tu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18023.txt.
Description
Zhuo Chen and co-authors teach vision-language models in 2025 to know what they do not know. These models falter on questions needing fresh or deep facts. Fetching outside help works but costs too much when done for every question. Their method fine-tunes a model on an automatically built set to flag when a question sits beyond its knowledge.
Tests on several visual question sets showed the boundary drawn well. Using it, the team cut needless fetching while keeping or raising accuracy. One model's boundary even served as a stand-in for others. Smarter fetching decisions could make such systems cheaper and faster in daily use.
Catalog record
| CREATOR | Zhuo Chen, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinyu Geng, Pengjun Xie, Fei Huang, Kewei Tu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-18023