Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training

Mingliang Liang, Zhuoran Liu, Arjen P. de Vries, Martha Larson · 2026

Read Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training by Mingliang Liang, Zhuoran Liu, Arjen P. de Vries, Martha Larson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27932.txt.

Introduction

Liang, Liu, de Vries, and Larson cut vision-language training cost by sampling data with DynamiCS, a cluster method that favors rare ideas. Past efficient training flattens topic mix but can starve rare concepts, while DynamiCS shrinks big clusters and grows small ones, keeping their rank order. It re-samples each epoch, widening cross-epoch variety and making the small-cluster boost workable. Tests show cheaper training plus stronger scores on long-tail ideas.

Catalog record

CREATORMingliang Liang, Zhuoran Liu, Arjen P. de Vries, Martha Larson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52524
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27932

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training Mingliang Liang, Zhuoran Liu, Arjen P. de Vries, Martha Larson Abstract The computational cost of training a vision-language model (VLM) can be reduced by sampling the training data. Previous work on efficient VLM pre-training has pointed to the importance of semantic data balance, adjusting the distribution of topics in the data to improve VLM accuracy. However, existing efficient pre-training approaches may disproportionately remove rare concepts from the training corpus. As a result, long-tail concepts remain insufficiently represented in the training data and are not effectively captured during training. In this work, we introduce a dynamic cluster-based sampling approach (DynamiCS) that downsamples large clusters of data and upsamples small ones. The approach is dynamic in that it applies sampling at each epoch. We first show the importance of dynamic sampling for VLM training. Then, we demonstrate the advantage of our cluster-scaling approach, which maintains the relative order of semantic clusters in the data and emphasizes the long-tail. This approach contrasts with current work, which focuses only on flattening the semantic distribution of the data. Our experiments show that DynamiCS reduces the computational cost of VLM training and provides a performance advantage for long-tail concepts. 1 Introduction Vision-Language Models (VLMs) demonstrate strong transferability [ 17 , 37 , 15 , 16 , 8 , 55 , 23 , 54 ] . Pre-trained VLMs can be applied to classification and image-text retrieval tasks, and their image encoders are widely used in Multimodal Large Language Models (MLLM) and generative models [ 18 , 27 , 33 , 38 ] . CLIP [ 37 ] , one of the popular VLMs, is trained on extremely large-scale datasets, requiring substantial GPU resources for pre-training. The training costs of CLIP have given rise to cost-saving training approaches, notably, RECLIP [ 19 ] , FLIP [ 23 ] , and CLIPA [ 22 ] , which maintain VLM performance but reduce training costs by reducing the amount of text or image information used from each sample for training. CLIP’s large-scale pre-training data is collected from the Web, but the data is curated to improve semantic data balance , i.e., to adjust the distribution of topics in the data, which in the wild typically has a very fat head and a very long tail. In [ 51 ] , the advantages of CLIP data curation are described as avoiding biases and balancing the data over the metadata, which corresponds to semantic categories. These advantages motivate the data curation approach in MetaCLIP [ 51 ] , which limits the number of data samples associated with each metadata category to 20k, effectively chopping off the fat head of the distribution. Figure 1: Zero-shot top-1 accuracy on ImageNet-1K [ 7 ] and Let-it-wag! [ 44 ] (long-tail test set). DynamiCS outperforms cost-saving baselines (RECLIP [ 19 ] , FLIP [ 23 ] , CLIPA [ 22 ] ) and dual-purpose approaches (DataComp [ 13 ] , DFN [ 12 ] , Captioning [ 31 ] ) while using less computational resources and achieves accuracy competitive with full-scale pre-training, e.g., OpenCLIP [ 15 ] . Experiments are conducted with ViT-B/16 pre-training of 6 epochs, applying different strategies to LAION-400M [ 40 ] or DataComp [ 13 ] . Dual-purpose approaches apply data sampling for the simultaneous purposes of training cost reduction and achieving semantic balance. This gap is left open by MetaCLIP [ 51 ] , whose goal is a semantically balanced training set. Metadata is not the only means of identifying semantic structure, rather clustering can be used instead, as with density-based pruning (DPB) [ 1 ] . Dual-purpose approaches can also combine data sampling for training cost reduction with sampling to improve data quality. For example, approaches have removed text-image pairs using another already-trained CLIP as a filter [ 13 , 12 ] and integrating information from automatically generated captions [ 31 ] . However, these approaches complicate the goal of training cost reduction since they all make critical use of another already-trained VLM. In this paper, we propose a dual-purpose approach for VLM training reduction, DynamiCS 1 1 1 https://github.com/MingliangLiang3/DynamiCS , which reduces costs with dynamic sampling and addresses semantic data balance with cluster scaling. In Fig. 1 , we observe that DynamiCS achieves a substantial speedup over OpenCLIP [ 15 ] , MetaCLIP [ 51 ] , and HQ-CLIP [ 49 ] , while also yielding a slight improvement on ImageNet-1K zero-shot classification. It improves (top of graph) over other cost-saving training approaches (RECLIP, CLIPA, FLIP) and other dual-purpose approaches (DataComp, DFN, Captioning) in terms of training cost and also performance. The improvements are particularly remarkable on long-tail test data (bottom of the graph), where the pre-trained DynamiCS ViT-B/16 models substantially outperform the baselines on the long-tail benchmark Let-it-wag! [ 44 ] classification task. The inspiration for DynamiCS is a change of perspective from previous semantic data balancing approaches, which “aim for even” distributions of pre-training data over semantic categories. The “aim for even” philosophy is represented by MetaCLIP [ 51 ] , which chops off the fat head, and DBP [ 1 ] , which homogenizes densities with a comparable effect. Cluster scaling instantiates an “aim for utility” philosophy, which does not necessarily result in a flattened distribution. Specifically, it downsamples the semantic fat head, without aiming to completely eliminate it. At the same time, it upsamples the semantic long tail, which improves the VLM performance on long-tail concepts. In sum, this paper makes the following contributions: • We establish that dynamic sampling makes an important contribution to efficient and long-tail-aware VLM pre-training when data sampling is used to reduce training costs. • We point out that data sampling should involve not only down- but also up-sampling, in order to improve VLM performance on the long tail concepts. • On the basis of these insights, we propose a dynamic cluster-based sampling approach (DynamiCS), which outperforms other cost-reducing VLM training approaches and is competitive with full-scale OpenCLIP [ 15 ] while requiring only about 3% of its training costs. The paper is structured as follows: in Sec. 2 , we cover related work. Then, we introduce DynamiCS (Sec. 3 ) and our experimental setup (Sec. 4 ). We provide further motivation with experimental analysis on a smaller-scale data set (Sec. 5 ). In Sec. 6 , we present experimental results on a large-scale dataset that demonstrate the advances of DynamiCS. Sec. 7 provides discussion and outlook. 2 Related Work Figure 2: Image examples from three semantic clusters (e.g., “sea,” “tennis” “dog”), where visually similar concepts are grouped together. 2.1 Vision-Language Models Large-scale Vision-Language Models (VLMs) have demonstrated remarkable transferability [ 17 , 37 , 15 , 16 , 8 , 55 , 23 , 54 ] , which the basis for their widespread applicability and effectiveness in downstream tasks. CLIP (Contrastive Language-Image Pre-Training) exemplifies this paradigm by learning visual–semantic embeddings from language through contrastive learning [ 37 , 45 ] . CLIP pre-training relies on large-scale datasets and can use millions or even billions of image–text pairs. Following CLIP [ 37 ] , subsequent work, such as OpenCLIP [ 15 ] and MetaCLIP [ 51 ] , has advanced

Read the complete text