No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models
Hai X. Pham, David T. Hoffmann, Ricardo Guerrero, Brais Martinez · 2026
Read No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models by Hai X. Pham, David T. Hoffmann, Ricardo Guerrero, Brais Martinez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25722.txt.
Introduction
Pham and co-authors improve how contrastive vision-language models learn composition. Past fixes used hard negative samples, which can hurt zero-shot skill. The authors trace the problem to long captions and to global pooling that drops binding details. Their fix aligns short, concept-focused caption parts with the image. A simple cross-modal attention pooling builds concept-focused visual features. They report top scores on compositionality tests. Zero-shot and retrieval skill stayed strong, with no added inference cost.
Catalog record
| CREATOR | Hai X. Pham, David T. Hoffmann, Ricardo Guerrero, Brais Martinez |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56089 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25722
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models Hai X. Pham, David T. Hoffmann, Ricardo Guerrero, Brais Martinez Abstract Contrastive vision-language (V&L) models remain a popular choice for various applications. However, several limitations have emerged, most notably the limited ability of V&L models to learn compositional representations. Prior methods often addressed this limitation by generating custom training data to obtain hard negative samples. Hard negatives have been shown to improve performance on compositionality tasks, but are often specific to a single benchmark, do not generalize, and can cause substantial degradation of basic V&L capabilities such as zero-shot or retrieval performance, rendering them impractical. In this work we follow a different approach. We identify two root causes that limit compositionality performance of V&Ls: 1) Long training captions do not require a compositional representation; and 2) The final global pooling in the text and image encoders lead to a complete loss of the necessary information to learn binding in the first place. As a remedy, we propose two simple solutions: 1) We obtain short concept centric caption parts using standard NLP software and align those with the image; and 2) We introduce a parameter-free cross-modal attention-pooling to obtain concept centric visual embeddings from the image encoder. With these two changes and simple auxiliary contrastive losses, we obtain SOTA performance on standard compositionality benchmarks, while maintaining or improving strong zero-shot and retrieval capabilities. This is achieved without increasing inference cost. We release the code for this work at https://github.com/SamsungLabs/concept_centric_clip . 1 Introduction (a) (b) Figure 1 : Method overview . (a) SigLIP uses a learnable query token in combination with an attention layer to pool the visual tokens into a single token. Aligning only global representations hampers the learning of a compositional representation. (b) Similar to SigLIP, our method aligns the global representations v and t . To simplify learning of a compositional representation, our method extends SigLIP by first, pooling the text encoder output tokens into concept embeddings \{c_{k}\} , which are used to attention-pool concept-specific information from the visual tokens \bar{V} , resulting in \hat{v}(c) . \hat{v}(c_{k}) and the corresponding c_{k} are aligned using L_{xac} . Furthermore, the global visual representation v is aligned with all c_{k} via {\mathcal{L}}_{\textbf{npc}} , a multi-positive variant of SigLIP loss. Similarly, the global image and text representations v and t are aligned via \mathcal{L}_{contrastive} . Contrastive vision-language (V&L) models have been a cornerstone of computer vision and machine learning since the publication of CLIP ( 25 ) . While contrastive V&L models have achieved remarkable performance on a large variety of tasks and enabled deployment of vision models to open world settings, various limitations have been found. Most notably, they struggle when learning compositional representations, such as binding a noun to its attributes, understanding the relations between objects, or recognizing when one of multiple objects has been replaced by another object 33 ; 10 ; 6 . Compositionality is a core capability of vision systems and results in various failure cases if not learned. Therefore, many works have focused on ways to improve compositionality post-hoc ( 33 ; 1 ; 8 ; 29 ; 28 ; 35 ; 4 ; 5 ; 23 ; 10 ; 15 ; 6 ) , most of which ( 28 ; 35 ; 4 ; 5 ; 23 ) rely on clever ways of constructing hard negatives for fine-tuning purposes. Previous works correctly identified one of the reasons why contrastive models do not learn compositional representations: A simple Bag-of-Words (BoW) representation is completely sufficient to retrieve the right image from a batch, as the caption is typically long and detailed. They have proposed solving this issue by training with hard negatives generated by following simple rules. Models trained on these negatives learn a compositional representation in the narrow domain defined by the set of rules used to generate hard negatives. Training with hard negatives has indeed been shown to yield in-distribution improvements, but these gains often fail to generalize to hard negatives that exhibit slightly different structures ( 10 ; 15 ) . At the same time, the improvements in compositionality achieved via hard negative training frequently come at the expense of generalization to other downstream tasks; precisely the property that originally made contrastive V&L models popular. Here, we follow a different approach and focus on better exploiting existing pre-training data. In particular, we focus on noun-phrases, which are groups of words that function as a concept in a sentence, e.g. “a red couch” or “the tall building”, and define two auxiliary loss terms, a contrastive concept loss and a cross-attentive pooling loss, which combine to rectify the root causes of the contrastive learning behavior. Our contrastive concept loss focuses on contrasting noun-phrases and images. Noun-phrases are short and thus are not solvable via a BoW representation. To provide an example: For the noun-phrase “a red couch”, a BoW representation is likely insufficient, as the probability of an image containing a couch and anything colored in red is quite high. Thus, the model must learn a latent representation that is more discriminative. Since our method relies on real data positives instead of synthetically generated hard negatives, our method is less likely to learn shortcuts that do not generalize to broader domains. Despite the shortcuts when training with long captions, the overall architectural design of contrastive V&L models also hinders the learning of a good compositional representation. Visual and textual representations are pooled into single-vector embeddings, which mix nouns and adjectives from different regions of the image or different parts of the caption. This global pooling operation results in a complete loss of associations between attributes and nouns. Fine-tuning with post-pooling hard negative losses can yield improvements by aligning the model to in-domain distributions, however, this will not substantially change the underlying behavior of the encoders. Instead of focusing on encouraging binding after the global pooling with hard negatives, we focus on learning binding before global pooling. Once learned, the model only needs to preserve a representation of that binding through the pooling operation. To encourage binding before the pooling operation, we introduce a cross-modal attention pooling function that extracts noun-phrase-specific information from the full visual representation, thereby producing multi-word, concept-specific embeddings learned with an auxiliary contrastive loss. As the attention-pooling function has no learnable parameters, this directly propagates the learning signal to the penultimate representation, i.e., it encourages a compositional representation before the global pooling operation. A schematic of our complete method can be seen in Fig. 1 . Our method, C 2 LIP (Concept-centric CLIP), is simple to use, comes with no additional trainable parameters, and requires only fine-tuning for a few epochs without the incurred cost of complicated hard negative generation pipelines. We show quantitatively that this simple method leads to improvements over its base model and reaches SOTA on the SugarCrepe 10 and SugarCrepe++ 6 benchmarks. At the same time, our method leads to improvements on image caption retrieval datasets while incurring only modest drops in classification performance on ImageNet. Note that the drop in performance on ImageNet can be attributed to two factors: First, fine-tuning is performed on a less diverse dataset than the one used