Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining

Raghuveer Thirukovalluru, Rui Meng, Karthikeyan K, Mingyi Su, Ping Nie, Semih Yavuz, Yingbo Zhou, Wenhu Chen, Bhuwan Dhingra, Duke University, Salesforce AI Research, Independent, University of Waterloo [email protected] · 2025

Read Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining by Raghuveer Thirukovalluru, Rui Meng, Karthikeyan K, Mingyi Su, Ping Nie, Semih Yavuz, Yingbo Zhou, Wenhu Chen, Bhuwan Dhingra, Duke University, Salesforce AI Research, Independent, University of Waterloo [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11293.txt.

Introduction

Thirukovalluru and co-authors present in 2025 a smarter way to build training batches for contrastive learning, the method that teaches embedding models by pulling like items together and pushing unlike ones apart. Batch size and quality drive how well such models learn, since each item's negatives come from its batchmates. Their method starts with a teacher model that ranks all examples.

From those ranks a sparse similarity graph is built, and a clustering step finds groups of items that make strong negatives for each other. Batches drawn from those groups are rich in hard negatives. On a large multi task benchmark, models trained this way set new records, and even small batches beat older results that needed batches many times larger.

Catalog record

CREATORRaghuveer Thirukovalluru, Rui Meng, Karthikeyan K, Mingyi Su, Ping Nie, Semih Yavuz, Yingbo Zhou, Wenhu Chen, Bhuwan Dhingra, Duke University, Salesforce AI Research, Independent, University of Waterloo [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11293

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining Raghuveer Thirukovalluru, Rui Meng, Karthikeyan K, Mingyi Su, Ping Nie, Semih Yavuz, Yingbo Zhou, Wenhu Chen, Bhuwan Dhingra, Duke University, Salesforce AI Research, Independent, University of Waterloo [email protected] Abstract Contrastive learning (CL) is a prevalent technique for training embedding models, which pulls semantically similar examples (positives) closer in the representation space while pushing dissimilar ones (negatives) further apart. A key source of negatives are “in-batch” examples, i.e., positives from other examples in the batch. Effectiveness of such models is hence strongly influenced by the size and quality of training batches. In this work, we propose Breaking the Batch Barrier ( B3 ), a novel batch construction strategy designed to curate high-quality batches for CL. Our approach begins by using a pretrained teacher embedding model to rank all examples in the dataset, from which a sparse similarity graph is constructed. A community detection algorithm is then applied to this graph to identify clusters of examples that serve as strong negatives for one another. The clusters are then used to construct batches that are rich in in-batch negatives. Empirical results on the MMEB multimodal embedding benchmark (36 tasks) demonstrate that our method sets a new state of the art, outperforming previous best methods by +1.3 and +2.9 points at the 7B and 2B model scales, respectively. Notably, models trained with B3 surpass existing state-of-the-art results even with a batch size as small as 64, which is 4–16× smaller than that required by other methods. 1 Introduction Contrastive Learning (CL) has emerged as the dominant approach for training embedding models [ 5 , 2 , 8 ] . It typically operates on data in the form of (query, positive) pairs [ 28 ] , where the objective is to minimize the distance between the query’s representation and that of its positive counterpart. To foster more discriminative representations, CL also incorporates negative examples – instances which, for a given query, are not its designated positive – by increasing their representational distance from the query. In practice, other examples within the same batch serve as readily available in-batch negatives [ 9 ] . Furthermore, to significantly enhance the learning signal, “hard negatives” are often explicitly mined and integrated into the training process [ 5 , 11 ] . These hard negatives are particularly challenging examples that share superficial features with the query or its positive, making them easily confusable with the true positive, yet are semantically distinct. Recent text-only embedding models, such as NV-Embed [ 11 ] and SFR-Embedding [ 18 ] , have achieved state-of-the-art results through the use of hard negative mining from the training dataset. These methods typically employ a pretrained teacher model to rank potential negatives relative to a given query, subsequently selecting a small set (e.g., up to ten) of top-ranked candidates as hard negatives. While incorporating multiple hard negatives can enhance model performance, it also substantially increases training cost and duration (e.g. using ten hard negatives is approximately 4x training time compared to using one hard negative). This performance-cost trade-off, while manageable for text-only models, becomes computationally prohibitive in multimodal contexts. In such settings, positive examples often include high-resolution images, significantly amplifying the processing overhead associated with each additional hard negative [ 29 ] . Moreover, contrastive learning relies on large datasets, which are already hard to fully use in multimodal training [ 14 , 23 ] . Adding many hard negatives makes this even more demanding, limiting scalability. Consequently, many recent efforts in multimodal embedding learning have largely avoided explicit hard negative mining across the entire training dataset. For instance, VLM2Vec [ 8 ] repurposes data from diverse tasks – such as retrieval, classification, and VQA – for contrastive training but omits dedicated hard negative sampling. MegaPairs [ 37 ] and mmE5 [ 1 ] focus on generating synthetic contrastive pairs, prioritizing data augmentation over mining challenging negatives. Other methods, including LLaVE [ 10 ] and UniLM [ 6 ] , do incorporate negative sampling strategies, but restrict this to resampling examples within the current batch as negatives. While these approaches have proven effective to varying degrees, they often do not exploit the potentially stronger negative signals residing outside the immediate batch, thereby overlooking a rich source of contrastive supervision. This work addresses the aforementioned gap by constructing batches from the full dataset where examples within each batch serve as strong negatives for one another. This approach aims to eliminate the need for separately mined hard negatives, significantly reducing computational cost. Our proposed batch mining method, B3 , leverages teacher-based ranking but revolutionizes batch construction. Unlike prior methods that add individually mined hard negatives to IID sampled batches, B3 uses graph-based community detection to cluster examples which are mutually strong negatives of each other. Batches are then strategically formed by sampling from these cohesive communities ensuring strong in-batch negatives. This facilitates strong training signals without the need for extra negatives. Our contributions are summarized as follows: - We introduce B3 , a novel batch mining strategy that sets a new state-of-the-art on the MMEB benchmark, surpassing strong baselines by 1.3 and 2.9 points at the 7B and 2B model scales, respectively. - We provide theoretical justification for the design of B3 . - We conduct a comprehensive ablation study to evaluate the individual components of B3 and highlight the significant impact of the batch mining module. - B3 achieves sota performance at 2B scale despite training at a much smaller batch size of 64. - B3 additionally significantly beats random batching baseline with strong hardnegatives while consuming only half the compute. - B3 does better than other batch mining techniques across multiple batch sizes. 2 Related Work This section reviews prior work on improving negative sampling for contrastive learning, strategies for batch mining, and recent advances in multimodal embedding methods. 2.1 Mining Better Negatives E5 [ 30 ] leveraged GPT-4 to construct contrastive learning (CL) datasets with diverse-length (anchor, positive, negative) tuples, which were then used to fine-tune the Mistral 7B model. SumCSE [ 27 ] used compositional transformations to create negatives. Gecko [ 12 ] employed a large LLM to generate queries from existing passages and subsequently relabeled positives and negatives for these queries. In contrast, our approach treats positives from other examples as potential negatives for a given query. Orthogonal to this, several recent studies, including SFR-Embedding [ 18 , 17 ] , NV-Embed [ 11 ] and NV-Retriever [ 20 ] , incorporate diverse annotated datasets — including clustering and classification — in addition to retrieval data, achieving notable gains on MTEB. A common technique in these methods involves using a pretrained teacher model to rank all positive examples from other queries in the dataset relative to a given query, selecting the top-ranked instances as hard negatives. Drawing inspiration from these studies, our approach also employs a teacher model to assess relationships between examples. However, instead of directly selecting and adding hard negatives, we leverage this ranking information to construct high-quality batches where the constituent examples inherently serve as strong negatives for one another during contrastive training. 2.2 Batch Mining for Contrastive Learning Several studies have explored improving contrastive models by optimizing the

Read the complete text