Half-Truths Break Similarity-Based Retrieval
Bora Kargi, Arnas Uselis, Seong Joon Oh · 2026
Read Half-Truths Break Similarity-Based Retrieval by Bora Kargi, Arnas Uselis, Seong Joon Oh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-23906.txt.
Description
Bora Kargi and two co-authors show that similarity-based image-text retrieval breaks under half-truths, descriptions extended with a plausible but wrong detail. The 2026 paper argues that adding a wrong detail should lower similarity, yet CLIP-style encoders often score the longer, partly false description higher.
On a standard caption set, CLIP picks the correct shorter description only a minority of the time, and it fares worse when the added detail is a relation rather than an object. The authors trace the flaw to contrastive training, which aligns whole sentences without forcing each entity and relation to be grounded. Their fix, CS-CLIP, splits captions into entity and relation units, builds a minimally edited foil for each, and fine-tunes the model to prefer the true unit. Half-truth accuracy rises sharply, with gains on compositional benchmarks too.
Catalog record
| CREATOR | Bora Kargi, Arnas Uselis, Seong Joon Oh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-23906