Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training
Mingliang Liang, Martha Larson · 2024
Read Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training by Mingliang Liang, Martha Larson (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16148.txt.
Introduction
Mingliang Liang and Martha Larson wrote this machine learning paper, published in 2024. It studies how hiding parts of the text during training affects vision-language models, which learn to link images with words.
Training these models is costly, so shrinking the text they see can help. Earlier work compared ways of masking text and crowned a syntax-based method best. Liang and Larson show that what matters is how each method changes the mix of common and rare words left visible. Rare words carry much of the signal. Their method, CLIPF, masks text based on word frequency on purpose. In tests it beat the syntax method, above all when few words were shown. They also found other methods overtake syntax masking given enough training rounds.
Catalog record
| CREATOR | Mingliang Liang, Martha Larson |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16148
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training Mingliang Liang, Martha Larson Abstract Vision Language Models (VLMs) can be trained more efficiently if training sets can be reduced in size. Recent work has shown the benefits of masking text during VLM training using a variety of strategies (truncation, random masking, block masking and syntax masking) and has reported syntax masking as the top performer. In this paper, we analyze the impact of different text masking strategies on the word frequency in the training data, and show that this impact is connected to model success. This finding motivates Contrastive Language-Image Pre-training with Word Frequency Masking (CLIPF), our proposed masking approach, which directly leverages word frequency. Extensive experiments demonstrate the advantages of CLIPF over syntax masking and other existing approaches, particularly when the number of input tokens decreases. We show that not only CLIPF, but also other existing masking strategies, outperform syntax masking when enough epochs are used during training, a finding of practical importance for selecting a text masking method for VLM training. Our code is available online. 1 Introduction Vision-Language Models (VLMs) learn embeddings that represent both visual and textual modalities simultaneously, and are known for the ease with which their strong performance transfers from one domain to another [ 10 , 28 , 32 , 16 , 30 , 8 , 47 , 22 ] . CLIP (Contrastive Language-Image Pre- training) [ 32 , 15 , 46 ] is a VLM that is trained on a very large number of image-text pairs collected online, where the text is considered a “natural language” annotation for the image. However, pre-training CLIP models typically requires thousands of GPU-days [ 32 , 15 ] . To accelerate the pre-training process, current work makes use of masking strategies for image and text tokens when pre-training CLIP models [ 22 , 43 , 20 ] . In addition to the speed-up, masking makes it possible to use large batch sizes, which enhances the benefits of contrastive learning and has been considered as a form of regularization, reducing the risk of overfitting [ 22 ] . For these reasons, despite the loss of some information from the original image and text, masking strategies have the potential to improve model performance. Figure 1 : Zero-shot classification accuracy on ImageNet-1K using models trained on CC12M to which different text masking strategies have been applied. The backbone of the image encoder is ViT-B/16. The text masking ratio is 75%. We also use image masking (75%) to speed up pre-training. At each point, we apply an additional epoch of training on the full, unmasked data. Currently, the state-of-the-art in text masking is represented by CLIPA [ 20 ] . The authors compare two strategies originally introduced by FLIP [ 22 ] , truncation and random word selection, and two new approaches, random block selection and syntax-based selection, i.e., masking on the basis of part-of-speech (POS) information. In [ 20 ] , syntax masking is found to deliver the highest performance. In this paper, we carry out an analysis of word frequency in the training data before and after text masking and discover a connection between the impact of a masking strategy on the training data and the success of a VLM. As shown in Fig. 1 , our experiments reveal that syntax masking falls behind other text masking strategies (random and block) as the number of epochs increases. Our experiments establish two important conclusions: First, syntax masking is not the best performing text masking approach, updating the state-of-the-art finding in [ 20 ] . Second, considering word frequency allows us to explain why random and block masking outperform truncation and syntax masking. This analysis motivates Contrastive Language-Image Pre-training with Word Frequency Masking (CLIPF), a new text masking strategy that directly leverages word frequency proposed in this paper. Because CLIPF leverages word-frequency directly, it does not need to expend compute on POS tagging, necessary for syntax masking. Fig. 1 shows that CLIPF outperforms syntax masking. CLIPF is inspired by earlier training of text models, that has leveraged frequency masking [ 29 ] . The key algorithmic novelty of CLIPF is the use of frequency-based word scores to prioritize words for masking, rather than to impose a hard decision on word removal. An important aspect of Fig. 1 is the development of performance over epochs. In contrast to the experiments in FLIP [ 22 ] and CLIPA [ 20 ] , we continue training until additional epochs yield only minimal gains. Fig. 1 demonstrates that our CLIPF approach is useful because it is the top performer in the range of 16-40 epochs, typically used in practice (e.g., OpenCLIP [ 15 ] trains on 30 epochs.) In sum, our paper makes the following contributions: • We show that considering the impact of text masking on word frequency in the training data differentiates between successful and less successful text masking strategies. • We introduce Contrastive Language-Image Pre-training with Word Frequency Masking (CLIPF), which leverages frequency directly during masking, and perform an in-depth quantitative comparison among existing masking strategies and CLIPF. • We point out the importance of the number of training epochs when comparing text masking strategies. Our quantified evaluation among existing alternative text masking approaches is carried out at two scales. A large-scale experiment on the LAION-400M [ 35 ] data set reproduces the CLIPA results and shows CLIPF outperforming syntax masking. A large number of smaller scale experiments on the CC3M [ 36 ] and CC12M [ 3 ] data sets, which are widely used for VLM research [ 15 , 21 , 30 , 43 , 18 ] , demonstrate that consideration of frequency is important in two ways. First, the frequency-based approach CLIPF emerges an excellent choice for text masking, especially as the number of input tokens decreases. Second, the text masking approach should be selected specifically for the application scenario (i.e., data set, input size, number of training epochs). However, in general masking strategies that disproportionately impact word frequency (truncation and syntax) should be avoided and strategies that protect relative word frequencies (random, block and CLIPF) should be preferred. Our code is available online. 1 1 1 https://github.com/MingliangLiang3/CLIPF 2 Related work 2.1 Vision-Language Models Vision-language models (VLMs) learn image embeddings from large-scale natural language supervision, which is important for making models transferable. CLIP [ 32 ] is the leading example of a VLM widely-used in practice, and other examples are ALIGN, SigLIP, and FLIP [ 32 , 16 , 45 , 22 ] , CLIP utilizes an image encoder and a text encoder to generate the representation of the image and text, and learn the joint representation by contrastive learning [ 32 , 39 ] . Because CLIP does not release its pre-training dataset, OpenCLIP pre-trained models on several large-scale public datasets [ 3 , 38 , 35 , 34 ] to reproduce CLIP. Although CLIP and OpenCLIP are pre-trained on very large-scale datasets, recent works like DeCLIP [ 21 ] and SLIP [ 30 ] demonstrate that Vision-language Models (VLMs) can be trained efficiently with smaller datasets, enabling researchers with limited computational resources to contribute. Moreover, masking image patches or reducing