FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Pretraining
Myunsoo Kim, Seong-Woong Shim, Byung-Jun Lee · 2025
Read FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Pretraining by Myunsoo Kim, Seong-Woong Shim, Byung-Jun Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11192.txt.
Introduction
Myunsoo Kim, Seong-Woong Shim, and Byung-Jun Lee present FALCON, a way to build training batches that dodge false negatives in vision language pretraining. In big sets many image text pairs match in truth, so a marked negative may be a true match in disguise. Those clash signals warp the learned space.
Rather than fixed rules, a scheduler picks negatives of the right hardness for each anchor as training goes. A proxy for match gain guides the picks. Gains held across three famed frames and many down tasks. Hard mining can thus stay sharp without poisoning itself.
Catalog record
| CREATOR | Myunsoo Kim, Seong-Woong Shim, Byung-Jun Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11192
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Pretraining Myunsoo Kim, Seong-Woong Shim, Byung-Jun Lee Abstract False negatives pose a critical challenge in vision-language pretraining (VLP) due to the many-to-many correspondence between images and texts in large-scale datasets. These false negatives introduce conflicting supervision signals that degrade the learned embedding space and diminish the effectiveness of hard negative sampling. In this paper, we propose FALCON (False-negative Aware Learning of COntrastive Negatives), a learning-based mini-batch construction strategy that adaptively balances the trade-off between hard and false negatives during VLP. Rather than relying on fixed heuristics, FALCON employs a negative mining scheduler that dynamically selects negative samples of appropriate hardness for each anchor instance during mini-batch construction, guided by a proxy for cross-modal alignment improvement. Experimental results demonstrate that FALCON significantly improves performance across two widely adopted VLP frameworks (ALBEF, BLIP-2) and a broad range of downstream tasks and evaluation settings, underscoring its effectiveness and robustness in mitigating the impact of false negatives. 1 Introduction Figure 1 : Illustration of semantic distance between an anchor text and multiple image samples in the vision-language embedding space. The goal of Vision-and-Language Pre-training (VLP) is to learn cross-modal representations from large-scale image-text pairs, improving performance on downstream tasks such as image-text retrieval (IRTR) frome2013devise , visual question answering (VQA) antol2015vqa , and natural language for visual reasoning (NLVR) suhr2018corpus . Recent advancements li2021align ; li2022blip ; li2023blip have demonstrated remarkable progress in this domain, underscoring the effectiveness of VLP in cross-modal representation learning. These models are typically trained with self-supervised objectives such as masked language modeling (MLM), image-text contrastive (ITC), and image-text matching (ITM) losses. While ITC and ITM are effective in improving the quality of the learned embedding space, they inherently require negative samples during training. In particular, the inclusion of hard negatives —those that are semantically similar to the anchor sample—has been shown to be crucial for the success of VLP schroff2015facenet . As a result, substantial research efforts have been devoted to devising more effective strategies for choosing such hard negatives li2021align ; li2022blip ; li2023blip ; byun2022grit . Although effective, these hard negative sampling strategies exhibit a critical limitation: as the selection criterion favors negatives with higher semantic similarity to the anchor, the likelihood of mistakenly identifying true positives as negatives (i.e., false negatives ) correspondingly increases byun2024mafa (see Figure 1 ). This challenge is particularly acute in VLP, where large-scale web-crawled datasets often exhibit many-to-many correspondences between images and text byun2024mafa ; jiang2023vision . Incorporating such false negatives into contrastive learning can substantially degrade representation quality by compelling the model to separate embeddings that should ideally remain close within the shared embedding space. To address this challenge, a few recent works attempted to mitigate this issue by leveraging strong pretrained models, either by refining the loss function to reduce the impact of false negative jiang2023vision ; radenovic2023filtering or relabeling false negatives as positives byun2024mafa ; bulat2024fff . However, these methods heavily depend on the reliability of the pretrained models and the heuristics used to identify false negatives, potentially limiting their generalizability across diverse datasets and training settings. In this paper, we present FALCON (False-negative Aware Learning of COntrastive Negatives), a learning-based mini-batch construction strategy for VLP that adaptively balances the trade-off between informative (hard) negatives and misleading (false) negatives. By explicitly optimizing this trade-off during mini-batch construction, FALCON effectively accounts for the inevitable presence of false negatives, improving the quality of learned cross-modal representations. Our empirical results demonstrate that FALCON significantly outperforms heuristic baselines across a wide range of downstream tasks and evaluation settings. Due to space constraints, the detailed discussion on related works is provided in the Appendix A . 2 Motivation In this section, we demonstrate that the false negative problem is an inherent and unavoidable challenge in VLP. We further motivate the need for a scheduling strategy that dynamically balances the trade-off between hard and false negatives throughout the VLP process. Inevitability of false negatives in VLP Given an image-text pair, the text may align with several plausible visual representations, and the image may correspond to multiple semantically valid textual descriptions. This intrinsic many-to-many alignment between visual and textual modalities leads to the unavoidable presence of false negatives in VLP. Proposition 1 . Let \mathcal{V}=\{v_{1},\ldots,v_{|\mathcal{V}|}\} denote the set of images and \mathcal{T}=\{t_{1},\ldots,t_{|\mathcal{T}|}\} the set of texts. Define \mathcal{R}\subseteq\mathcal{V}\times\mathcal{T} as the binary relation of semantic compatibility within the Cartesian product of image and text set, where (v,t)\in\mathcal{R} implies that t is a valid description of v . Let \mathcal{P}\subset\mathcal{R} denote the set of ground-truth positive pairs available to the learner, such that any pair (v,t)\notin\mathcal{P} is treated as negative during vision-language pretraining (VLP). Then, the probability of sampling a false negative remains strictly positive unless \mathcal{P}=\mathcal{R} , an assumption that is infeasible in practice. The proof is provided in the Appendix B . Note that the theoretical corner case \mathcal{P}=\mathcal{R} , which corresponds to a setting where all semantically compatible image-text pairs are known and labeled as positives, is infeasible in practice as soon as we account for the presence of synonyms, paraphrases, and stylistic or distributional variations introduced by human or AI-generated annotations parashar2024neglected ; urbanek2024picture ; wang2024picture ; momeni2023verbs . These factors render the space of valid image-text alignments effectively infinite, making the complete manual identification and exclusion of false negatives impractical. Figure 2 : (Top Left) Risk of false negatives across training steps for different similarity quantiles q used in mini-batch construction during VLP byun2022grit . False negatives are identified using the pretrained BLIP-129M model li2022blip . (Bottom Left) False negative ratio during VLP when false negative filtering is applied using the pretrained model’s ITM score at varying thresholds. On the right, Text-to-Image (Top Right) and Image-to-Text (Bottom Right) examples are shown where the pretrained model fails to assign high ITM scores to false negatives. Instances highlighted in red represent false negatives. Optimal similarity is anchor-specific and evolves over training To retrieve hard negatives, prior works have commonly adopted cosine similarity of embeddings as a selection criterion li2021align ; li2022blip ; li2023blip ; byun2022grit ; byun2024mafa , selecting images or texts with high similarity to the anchor. It is widely acknowledged that there exists an optimal similarity range that facilitates effective hard negative mining while minimizing the risk of false negative inclusion wu2020conditional ; robinson2020contrastive . However, this optimal range is inherently anchor-specific and evolves throughout the training process, making it impractical to capture with a single fixed similarity level. In particular, semantically complex anchors are more difficult to align and are therefore learned more slowly, resulting in less mature and noisier embeddings even in later stages of training. For such anchors, or during the early phases of training, selection of negatives with sufficiently low similarity is necessary to reduce the risk of false negatives. In contrast, semantically simple anchors are aligned more rapidly and yield well-formed embeddings earlier in training. In these cases, or in the later stage of training, the similarity distribution of positive pairs becomes tighter and concentrated at higher values, permitting the safe mining of more similar negatives. Consequently, VLP methods that rely on embedding similarity are exposed to a dynamic, anchor-dependent risk of false negative selection that evolves throughout