Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models

Dhruba Ghosh, Yuhui Zhang, Ludwig Schmidt, Stanford University · 2026

Read Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models by Dhruba Ghosh, Yuhui Zhang, Ludwig Schmidt, Stanford University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17871.txt.

Description

Dhruba Ghosh and team test the fine-grained knowledge of vision-language models, a weak spot behind strong benchmark scores. Vision-language models have advanced across visual question answering, visual reasoning, document understanding and multimodal dialogue, built on varied base models, alignment designs and training data. Yet recent work shows they trail on traditional image classification benchmarks that test fine-grained visual knowledge.

The team tests many recent models on fine-grained classification benchmarks and hunts for factors behind the gap between fine-grained knowledge and other vision scores. Ablation tests show a better language model lifts all benchmark scores equally, while a better vision encoder lifts fine-grained classification most. The pretraining stage also proves vital, above all when language model weights stay unfrozen during pretraining. Those insights point toward stronger fine-grained visual understanding and vision-centric skills.

Catalog record

CREATORDhruba Ghosh, Yuhui Zhang, Ludwig Schmidt, Stanford University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-17871

Browse the full library · About the Tower of Babel