Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding
Eun Woo Im, Dhruv Madhwal, Vivek Gupta Arizona State University · 2026
Read Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding by Eun Woo Im, Dhruv Madhwal, Vivek Gupta Arizona State University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13313.txt.
Introduction
Im, Madhwal, and Gupta improve compositional understanding in vision-language models by mining harder contrastive negatives. Their ConcretePlant method targets highly concrete words, whose changes create sharper visual and structural contrasts, and their Cement loss uses a margin to rebalance gradients that easy pairs otherwise dominate. The combined Slipform framework reaches state-of-the-art accuracy across compositional benchmarks and also strengthens general cross-modal retrieval and linear probing.
Catalog record
| CREATOR | Eun Woo Im, Dhruv Madhwal, Vivek Gupta Arizona State University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38635 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.13313
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding Eun Woo Im, Dhruv Madhwal, Vivek Gupta Arizona State University Abstract \setheadertext Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding \resource Contact: {eunwooim, dmadhwal, vgupt140}@asu.edu Project page: https://eunwooim.github.io/concrete-jungle 1 Introduction Vision-Language Models (VLMs) have achieved remarkable performance across diverse multimodal understanding benchmarks [ 46 , 67 ] and have been well-established as representation modules in multimodal learning, retrieval, and generation. Throughout contrastive pretraining on web-scale image-caption datasets [ 51 , 54 , 50 , 18 , 10 , 59 ] within a shared embedding space, VLMs learn holistic and robust cross-modal representations. Despite these capabilities, VLMs are prone to biases stemming from limitations of the contrastive pretraining mechanism. These include failures to understand negations [ 2 ] , object counting [ 43 ] , spatial relations [ 72 ] and accurate entity associations [ 1 ] . One of the most significant limitations is bag-of-words-like behavior, which manifests as an inability to understand word order [ 55 , 31 ] or attribute binding [ 47 ] . This limitation is primarily a byproduct of the contrastive objective. In the standard multimodal contrastive pretraining, training batches are curated by randomly sampled data points from the training dataset corpus. However, this random sampling often fails to provide effective negative samples to differentiate compositional semantics for each data point. Therefore, VLMs tend to match global summaries rather than the grammatical structure of the caption or the fine-grained details of the scene. As a result, they may recognize the presence of objects while overlooking the underlying logic and relational bindings. Recent studies demonstrate that hard negative mining and the inclusion of these mined samples in the training batch can significantly mitigate this problem and improve compositional understanding [ 67 , 70 ] . Hard negatives are defined as misaligned image-caption pairs nevertheless overlaps the semantics significantly with the corresponding positive samples. Training on these pairs forces the model to differentiate between subtly distinct inputs. Hard negative generation techniques have evolved from simple text swapping using Part-Of-Speech (POS) tagging [ 21 , 66 ] and the application of handcrafted heuristics for spatial relationship augmentation [ 69 ] , to the utilization of Large Language Models (LLMs) [ 16 ] and image editing or generative models [ 45 ] . However, hard negative mining fundamentally samples from a one-to-many distribution of plausible compositional variations. State-of-the-art methods typically design the hard negative generator by prompting LLMs to perturb a single keyword within the caption [ 16 , 45 ] . Consequently, the generation process may vary and can introduce quality difference depending on the keyword selected by the LLM. This brittleness raises a research question: How can we maximize the quality of the hard negative samples and improve the effectiveness of those synthetic data? Rather than designing a better generator structure, we shift the focus to identifying the data quality factor and provide a solution to better utilize the generated data. This study shows that the lexical concreteness of a selected keyword is directly related to the structural discrepancy of the resulting hard negatives, and introduces a method to control this factor. Lexical concreteness [ 57 , 42 , 20 ] how strongly a concept is associated with perceptible entities or experiences. Focusing on highly concrete entities encourages the generative model to modify a localized physical concept rather than to make an ambiguous global transformation. We then compare the datasets generated with controlled levels of concreteness to verify the central hypothesis. Furthermore, we identify that the gradient imbalance increases substantially with batch size and show that it remains vulnerable even in small-batch settings. To mitigate this problem, a simple margin regularization is proposed with theoretical and empirical demonstration of its controllability. Analysis shows that logit gaps are correlated with keyword concreteness, and that the data generation process introduces a skewed prior over concreteness scores. Based on these observations, the margin is modeled with a Fermi-Dirac distribution to adaptively determine its value and balance gradient magnitudes during the training process. To this end, we introduce Concrete Jungle , a concreteness-aware pretraining method that includes the following contributions: (1) We identify the critical role of lexical concreteness in hard negative mining. Building on this, we propose ConcretePlant , an automated hard negative generation method that explicitly targets concrete concepts to curate more effective hard negatives. (2) We reveal the existence of gradient magnitude polarity even in small-batch settings. To address this, we introduce Cement loss , a simple yet effective loss function that utilizes an adaptive margin to regulate gradient magnitudes. (3) Through extensive analysis and experiments, concreteness is shown to be a statistically significant factor in training data, and the proposed method is demonstrated to achieve superior performance across benchmarks. 2 Maximizing Hard Negative Efficacy 2.1 Concreteness Matters in Hard Negative Mining Figure 1 : Examples of our main hypothesis. Perturbing abstract keywords yields minor visual changes, whereas concrete words produce larger structural changes. Suppose that an anchor image-caption pair (v_{i},t_{i}) is sampled from the data corpus \mathcal{D} . A standard hard negative generation pipeline typically prompts a LLM to identify a semantic keyword in the given caption t_{i} to generate a textual perturbation t_{i}^{\prime} . Subsequently, a hard negative image v_{i}^{\prime} is obtained by feeding v_{i} and t_{i}^{\prime} to an image generative model [ 45 ] . As shown in Fig. 1 , visual perceptibility of the resulting samples depends on how perceptible the keywords are. Perturbing perceptible keywords results in structural difference, while modifying abstract stylistic descriptor yields minimal visual distinction. We hypothesize that this indistinguishable variation between the v_{i} and v_{i}^{\prime} fails to provide an effective learning signal. This observation establishes the core hypothesis of the proposed methodology: psycholinguistic concreteness score c_{i} [ 7 ] of a selected text token is strongly dependent on the physical difference of the resulting hard negative image. Maximizing this metric during the keyword selection process systematically guarantees the visual semantic perturbation for cross-modal representation learning. Based on this idea, the following section explains a method for concreteness-aware hard negative mining. 2.2 ConcretePlant: Automating Concrete Negative Generation Figure 2 : Overview of ConcretePlant. The pipeline selects a concrete target word from the caption, generates a hard negative caption with Qwen3-32B, and edits the image with SDXL-Turbo to produce a hard negative pair. Concreteness-Aware Keyword Sampling Given an image-caption sample (v_{i},t_{i}) in source data corpus \mathcal{D} , the longest caption is selected to ensure semantic richness [ 14 , 32 ] if \mathcal{D} offers multiple caption pairs. t_{i} is decomposed into n semantic keywords t_{i}=\{t_{i}^{k}\}_{k=1}^{n} with SpaCy [ 21 ] , and corresponding concreteness scores c_{i}^{k}=\text{lookup}(t_{i}^{k},\mathcal{R}) are looked up from concreteness ratings database \mathcal{R} [ 7 ] . Top-K sampling is first performed to ensure the preference towards higher concreteness scores, then the lexical concreteness score c_{i}=\text{sample}(\text{top-K}(c_{i}^{1:n})) is defined by simple rule based sampling to balance attribute , object , relation (ARO) compositional categories. Additional details are referred to Appendix A.1 . Hard Negative Mining Data generation employs two generative models: Qwen3-32B [ 62 ] as the hard negative captioner f_{\text{cap}} , and SDXL-Turbo [ 49 ] for hard negative image generator f_{\text{gen}} . Given an anchor sample (v_{i},t_{i},c_{i})