Is CLIP ideal? No. Can we fix it? Yes!

Raphi Kang, Yue Song, Gerogia Gkioxari, Pietro Perona · 2025

Read Is CLIP ideal? No. Can we fix it? Yes! by Raphi Kang, Yue Song, Gerogia Gkioxari, Pietro Perona (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08723.txt.

Introduction

Raphi Kang and co-authors prove that CLIP's shared image-text space has a geometric flaw no training can fix, then propose a scoring method that routes around it. CLIP is the workhorse aligning pictures and words, yet it fumbles bindings, positions, and negation. Their 2025 paper asks whether the fault lies in the very idea of one joint embedding.

The proof says yes: no such space can simultaneously handle plain content plus any two of attribute binding, spatial relations, and negation. Their fix scores matches with dense cosine similarity maps, keeping each image patch and each word in play instead of crushing them into single vectors. Scores improve and stay interpretable. Vision-language researchers get a limit theorem and a practical patch in one paper.

Catalog record

CREATORRaphi Kang, Yue Song, Gerogia Gkioxari, Pietro Perona
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08723

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Is CLIP ideal? No. Can we fix it? Yes! Raphi Kang, Yue Song, Gerogia Gkioxari, Pietro Perona Abstract Contrastive Language-Image Pre-Training (CLIP) is a popular method for learning multimodal latent spaces with well-organized semantics. Despite its wide range of applications, CLIP’s latent space is known to fail at handling complex visual-textual interactions. Recent works attempt to address its shortcomings with data-centric or algorithmic approaches. But what if the problem is more fundamental, and lies in the geometry of CLIP? Toward this end, we rigorously analyze CLIP’s latent space properties, and prove that no CLIP-like joint embedding space exists which can correctly do any two of the following at the same time: 1. represent basic descriptions and image content, 2. represent attribute binding, 3. represent spatial location and relationships, 4. represent negation. Informed by this analysis, we propose Dense Cosine Similarity Maps (DCSMs) as a principled and interpretable scoring method for CLIP-like models, which solves the fundamental limitations of CLIP by retaining the semantic topology of the image patches and text tokens. This method improves upon the performance of classical CLIP-like joint encoder models on a wide array of benchmarks. We share our code and data here for reproducibility: https://github.com/Raphoo/DCSM_Ideal_CLIP 1 Introduction Contrastive Language-Image Pre-Training (CLIP) is a widely used method for pretraining vision language model (VLM) embeddings in downstream applications [ 49 ] . It jointly trains an image and text encoder, mapping both modalities into a shared latent space. In this space, cosine similarity between embeddings reflects semantic similarity between the text and image: a high value indicates a semantic match between text and image, while a low value suggests that they are unrelated. CLIP is simple, computationally efficient, and CLIP-based zero-shot multimodal tasks such as image classification and text-based retrieval can be impressively accurate. Unlike autoregressive models where an increasingly large number of images and texts to compare requires prohibitively longer context windows, CLIP can process each image and text prompt separately and only requires a simple inner product calculation for scoring. As a result, the list of systems that use CLIP embeddings toward more complex capabilities, such as image captioning [ 25 ] , visual question answering [ 54 ] , and text-guided image manipulation/generation [ 20 , 40 ] , gets longer by the day. Figure 1 : CLIP scores do not accurately reflect semantics of text prompts due to inherent geometric limitations. For five out of six pairs of captions, the incorrect text has the higher CLIP score with the image. By contrast, our proposed Dense Cosine Similarity Maps (DCSM) correctly scores matched pairs. The similarity score is unnormalized because it is predicted by a neural network. CLIP scores computed with OpenAI-CLIP ViT-B/32. CLIP, however, is not perfect. It struggles with spatial reasoning and compositional understanding [ 69 , 19 , 59 ] , concept and attribute binding [ 23 , 39 , 8 ] , as well as negation [ 1 , 55 ] , to name a few shortcomings. Examples in Fig. 1 demonstrate these failure modes. These defects of CLIP impact downstream models and tasks. For example, search engines which use CLIP will bring up images of yellow coats to the prompt “not a yellow coat", and CLIP-based generative models cannot reflect spatial relationships in the text prompt. Ever since CLIP’s conception, the vision-language community has worked to improve its semantics via adjustments in the training data [ 69 , 63 , 55 , 1 , 42 , 48 , 21 , 62 , 30 ] or the architecture and training procedure [ 27 , 28 , 53 , 9 , 61 , 37 , 72 ] . But could we be pushing Sisyphus’s boulder? In this work, we suggest to take a step back and reassess CLIP’s basic geometry and philosophy from first principles. We ask two questions: (1) Is it possible for there to exist a CLIP-like latent space which has all the properties necessary to succeed in popular VLM tasks, using cosine similarity as the reference metric? (2) If no such space exists - is there a way to “rescue" this latent space without forsaking CLIP altogether? To address these queries, we first formalize the geometry of the CLIP latent space and establish a list of conditions which must be satisfied in order for it to understand the precise meaning of images and texts, as required by popular VLM benchmarks [ 59 , 69 , 19 , 23 , 1 , 16 ] . Second, we show that these conditions are not achievable when using cosine similarity on the unit hypersphere. This answers the first question: an ideal CLIP space cannot exist. Finally, we propose a lightweight downstream CNN which utilizes a topological map relying on cosine similarity scores between each text token and image patch embedding, to produce a more perceptive text-image distance score in lieu of naive cosine similarity in CLIP-space. In summary, our contributions are: • Problem Identification : We find that naive cosine similarity on unit vector embeddings have fundamental geometric restrictions preventing it from accurately representing (1) attribute binding to distinct object concepts, (2) spatial relationships and location, and (3) negation. • Analysis/Proof : We define the CLIP latent space as a projection of images and texts which are composed of atomic objects, attributes, and spatial relationships, and formalize desired conditions for the latent space in order for it to succeed on existing VLM benchmarks. We prove for each condition that no satisfactory vector space exists. • Topology as a Solution : We propose to use a Dense Cosine Similarity Map (DCSM) from pre-trained CLIP to produce a more comprehensive text-image score. We present a prototype with a simple two-layer CNN module. • Experimental evaluation : We evaluate our method on multiple benchmark regimes against CLIP-like models, and observe consistent performance gain across tasks. 2 Prior Work Vision Language Models. Building upon the principles of large-scale contrastive pretraining and joint representation learning, recent CLIP-like VLMs [ 17 , 67 , 67 , 38 , 26 , 68 ] have significantly bridged the gap between vision and language. Beyond CLIP models, autoregressive VLMs [ 33 , 34 ] have emerged as compelling alternatives by jointly attending to both text and image embeddings. Neurosymbolic program synthesis methods [ 57 , 15 , 36 ] also mitigate some of CLIP’s limitations by formulating text-image semantic distance acquisition as several sub-problems. While these methods are more comprehensive than CLIP in complex visual reasoning, most of them rely on a latent space of CLIP-like models as a core submodule. As such, there remains a strong motivation to continue improving CLIP architectures by addressing their inherent shortcomings. Empirical Limitations of CLIP. A growing body of work has revealed several limitations of CLIP in handling complex visual-text interactions. Specifically, CLIP struggles with accurately binding attributes to concepts in multi-object scenes [ 23 , 39 , 8 ] , exhibits misinterpretations of object layouts or conflates multiple entities within a single scene [ 69 , 23 , 39 ] , and has difficulty in accurately representing negation [ 1 , 55 ] . To systematically evaluate these limitations, various vision language benchmarks have been proposed targeting different tasks, including attributes, relationships, and

Read the complete text