HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
Esra Dönmez, Pascal Tilli, Hsiu-Yu Yang, Thang Vu, Carina SilbererInstitute for Natural Language Processing, University of Stuttgart {esra.doenmez, pascal.tilli, hsiu-yu.yang, thang.vu, carina.silberer} @ims.uni-stuttgart.de · 2026
Read HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities by Esra Dönmez, Pascal Tilli, Hsiu-Yu Yang, Thang Vu, Carina SilbererInstitute for Natural Language Processing, University of Stuttgart {esra.doenmez, pascal.tilli, hsiu-yu.yang, thang.vu, carina.silberer} @ims.uni-stuttgart.de (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06157.txt.
Description
Esra Dönmez and four co-authors teach vision language models finer cross modal understanding with hard negative captions. Image text matching learns from web pairs whose captions often fit loosely, so models miss subtle mismatches between image and text. A model may accept a caption that swaps a detail, since training rarely shows such near miss cases.
Their data set, Hard Negative Captions, is built automatically with foiled captions that differ from true ones in small but meaning changing ways. They also provide a hand made test set with mismatch cases of rising compositional complexity. Training on the new data improved zero shot detection of mismatches on diagnostic tasks. Models also held up better under noisy visual input and served as a strong start for fine tuning.
Catalog record
| CREATOR | Esra Dönmez, Pascal Tilli, Hsiu-Yu Yang, Thang Vu, Carina SilbererInstitute for Natural Language Processing, University of Stuttgart {esra.doenmez, pascal.tilli, hsiu-yu.yang, thang.vu, carina.silberer} @ims.uni-stuttgart.de |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06157