Half-Truths Break Similarity-Based Retrieval

Bora Kargi, Arnas Uselis, Seong Joon Oh · 2026

Read Half-Truths Break Similarity-Based Retrieval by Bora Kargi, Arnas Uselis, Seong Joon Oh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-23906.txt.

Description

Bora Kargi and two co-authors show that similarity-based image-text retrieval breaks under half-truths, descriptions extended with a plausible but wrong detail. The 2026 paper argues that adding a wrong detail should lower similarity, yet CLIP-style encoders often score the longer, partly false description higher.

On a standard caption set, CLIP picks the correct shorter description only a minority of the time, and it fares worse when the added detail is a relation rather than an object. The authors trace the flaw to contrastive training, which aligns whole sentences without forcing each entity and relation to be grounded. Their fix, CS-CLIP, splits captions into entity and relation units, builds a minimally edited foil for each, and fine-tunes the model to prefer the true unit. Half-truth accuracy rises sharply, with gains on compositional benchmarks too.

Catalog record

CREATORBora Kargi, Arnas Uselis, Seong Joon Oh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-23906

Browse the full library · About the Tower of Babel