HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities

Esra Dönmez, Pascal Tilli, Hsiu-Yu Yang, Thang Vu, Carina SilbererInstitute for Natural Language Processing, University of Stuttgart {esra.doenmez, pascal.tilli, hsiu-yu.yang, thang.vu, carina.silberer} @ims.uni-stuttgart.de · 2026

Read HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities by Esra Dönmez, Pascal Tilli, Hsiu-Yu Yang, Thang Vu, Carina SilbererInstitute for Natural Language Processing, University of Stuttgart {esra.doenmez, pascal.tilli, hsiu-yu.yang, thang.vu, carina.silberer} @ims.uni-stuttgart.de (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06157.txt.

Description

Esra Dönmez and four co-authors teach vision language models finer cross modal understanding with hard negative captions. Image text matching learns from web pairs whose captions often fit loosely, so models miss subtle mismatches between image and text. A model may accept a caption that swaps a detail, since training rarely shows such near miss cases.

Their data set, Hard Negative Captions, is built automatically with foiled captions that differ from true ones in small but meaning changing ways. They also provide a hand made test set with mismatch cases of rising compositional complexity. Training on the new data improved zero shot detection of mismatches on diagnostic tasks. Models also held up better under noisy visual input and served as a strong start for fine tuning.

Catalog record

CREATOREsra Dönmez, Pascal Tilli, Hsiu-Yu Yang, Thang Vu, Carina SilbererInstitute for Natural Language Processing, University of Stuttgart {esra.doenmez, pascal.tilli, hsiu-yu.yang, thang.vu, carina.silberer} @ims.uni-stuttgart.de
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06157

Browse the full library · About the Tower of Babel