Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Models
Helen Qu, Sang Michael Xie · 2025
Read Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Models by Helen Qu, Sang Michael Xie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08000.txt.
Introduction
Helen Qu and Sang Michael Xie study how pairs of words in training data shape what image-text models can do. Their paper is from two thousand twenty-five. Models such as CLIP do best on ideas seen often in training. Less clear is what happens when a common thing shows up in a rare pairing with an other thing.
They gauge pair frequency with a score that strips out how common each word is alone. On made-up images, that score tracks test scores very closely. The same link shows in edited real photos, and it carries to larger models built on CLIP. Their point is that new methods are needed, since more paired data cannot cover all pairs.
Catalog record
| CREATOR | Helen Qu, Sang Michael Xie |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08000
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Models Helen Qu, Sang Michael Xie Abstract CLIP and large multimodal models (LMMs) have better accuracy on examples involving concepts that are highly represented in the training data. However, the role of concept combinations in the training data on compositional generalization is largely unclear – for instance, how does accuracy vary when a common object appears in an uncommon pairing with another object? In this paper, we investigate how word co-occurrence statistics in the pretraining dataset (a proxy for co-occurrence of visual concepts) impacts CLIP/LMM performance. To disentangle the effects of word co-occurrence frequencies from single-word frequencies, we measure co-occurrence with pointwise mutual information (PMI), which normalizes the joint probability of two words co-occurring by the probability of co-occurring independently. Using synthetically generated images with a variety of concept pairs, we show a strong correlation between PMI in the CLIP pretraining data and zero-shot accuracy in CLIP models trained on LAION-400M ( r = 0.97 r=0.97 and 14% accuracy gap between images in the top and bottom 5% of PMI values), demonstrating that even accuracy on common concepts is affected by the combination of concepts in the image. Leveraging this finding, we reproduce this effect in natural images by editing them to contain pairs with varying PMI, resulting in a correlation of r = 0.75 r=0.75 . Finally, we demonstrate that this behavior in CLIP transfers to LMMs built on top of CLIP ( r = 0.70 r=0.70 for TextVQA, r = 0.62 r=0.62 for VQAv2). Our findings highlight the need for algorithms and architectures that improve compositional generalization in multimodal models without scaling the training data combinatorially. Our code is available at https://github.com/helenqu/multimodal-pretraining-pmi . 1 Introduction Contrastive image-text encoders such as CLIP [ 1 , 2 ] are a crucial component of large multimodal models (LMMs) [ 3 , 4 , 5 , 6 ] that can perform a diverse array of vision-language tasks. CLIP’s strong zero-shot accuracy on challenging datasets, such as ImageNet-R and ObjectNet [ 7 , 8 , 9 ] , indicates that a large, diverse pretraining dataset may be sufficient for robust generalization [ 10 ] . To understand this, recent work has shown that CLIP accuracy on examples with a particular visual concept correlates strongly with the frequency of that concept in the pretraining data distribution [ 11 , 12 ] . Thus, a natural way forward is to scale the pretraining dataset for sufficient coverage on all concepts. However, extending this idea to concept combinations suggests that we need combinatorially many pretraining examples to generalize to all concept combinations, which is prohibitively expensive. In general, the role of concept combinations in pretraining data and its effects on accuracy in both CLIP and LMMs built on CLIP is underexplored. Prior compositional generalization benchmarks typically only test combinations unseen during pretraining [ 13 , 14 ] or are generic evaluations that do not consider the role of pretraining data [ 15 , 16 , 17 , 18 ] . In this work, we address this gap by investigating CLIP and LMM accuracies through the lens of the co-occurrence rate of concept pairs in the pretraining dataset. In particular, we focus on co-occurrence between words in the textual captions of CLIP pretraining examples as a proxy for visual concepts. To decorrelate the pair frequency from single-concept frequencies (i.e., the individual concepts in low frequency pairs are often themselves low frequency), we calculate pointwise mutual information (PMI) [ 19 ] for all concept pairs, which measures the probability of concept co-occurrence normalized by the expected probability if the concepts were independent. Figure 1: Overview of our contributions. (left) We extract concept pairs from pretraining data caption text and calculate their co-occurrence frequency and pointwise mutual information (PMI) across all captions in the dataset, including pairs that do not co-occur in the dataset at all. (middle) We design a synthetic dataset with concept combinations across a wide range of PMI and find a strong correlation between CLIP zero-shot classification accuracy and PMI (i.e., the same concept “broom” is less accurately identified in a low vs. high PMI pair). (right) We continue to find an accuracy vs. PMI correlation in large multimodal models built on CLIP embeddings evaluated on visual question-answering tasks. In this case, per-example PMI is averaged over all the concept pairs in the question-answer pair (shown in blue highlights). In our experiments, we find a strong correlation between concept pair PMI in the pretraining data and accuracy: in leading CLIP models trained on LAION-400M evaluated on zero-shot image classification, as well as in LMMs built on CLIP embeddings evaluated on visual question-answering. We test this by generating synthetic images with a variety of concept pairs, finding a 14% absolute (30% relative) difference in zero-shot classification accuracy between images in the bottom vs. top 5% of concept pair PMI. With this understanding, we reproduce this correlation in natural images by simply pasting in a small image of an object that rarely co-occurs in pretraining with the main image object (low PMI), reducing CLIP zero-shot classification accuracy by up to 10%. Finally, we evaluate LLaVA, a leading LMM built on CLIP, and find that the correlation between PMI and task accuracy still holds on visual question-answering benchmarks. This result even extends to LMMs built on closed-source embedding models such as OpenAI’s CLIP. Co-occurrence statistics in the pretraining data also lead to model biases. In particular, we find that LLaVA exhibits an output bias that is correlated with co-occurrence, tending to output “Yes” for questions with highly co-occurring concepts regardless of the true label. Overall, our findings show that CLIP and LMMs have biases that depend heavily on co-occurrence statistics from pretraining, rather than an understanding of the individual concepts. Our results point to the need for new algorithms and architectures that improve compositional generalization capabilities in multimodal models. 2 Setup We introduce the models and pretraining dataset used in our analyses. CLIP. Contrastive Language-Image Pretraining (CLIP) [ 1 ] is a self-supervised learning method that uses natural language supervision in the form of image captions to learn downstream task-agnostic image representations. Formally, in a batch of N image-text pairs \{(x_{i},t_{i})\}^{N}_{i=1} where x_{i}\in\mathcal{X} , t_{i}\sim\mathcal{T} , CLIP simultaneously trains an image encoder \phi:\mathcal{X}\rightarrow\mathcal{Z}_{v} and text encoder \psi:\mathcal{T}\rightarrow\mathcal{Z}_{t} , where \mathcal{Z}_{v}\subset\mathbb{R}^{d} , \mathcal{Z}_{t}\subset\mathbb{R}^{d} denote the image and text embedding spaces, respectively. The encoders are trained to minimize the multi-class N -pair loss: \displaystyle\ell_{\text{CLIP}}(\phi,\psi)=-\frac{1}{N}\sum_{i}\ln\frac{e^{\phi(x_{i})^{\top}\psi(t_{i})/T}}{\sum_{j}e^{\phi(x_{i})^{\top}\psi(t_{j})/T}}-\frac{1}{N}\sum_{j}\ln\frac{e^{\phi(x_{j})^{\top}\psi(t_{j})/T}}{\sum_{i}e^{\phi(x_{i})^{\top}\psi(t_{j})/T}}. (1) For a test image x , we can use the learned encoders for zero-shot classification by translating a list of classification label texts y_{1},\dots,y_{k} (where k is the number of classes) into pseudo-captions y\textquoteright_{1},\dots,y^{\prime}_{k} , e.g., a photo of {class name} , and selecting the class whose pseudo-caption embedding aligns best with the image embedding: \argmax_{i}\phi(x)^{\top}\psi(y\textquoteright_{i}) . LMMs. Large multimodal models (LMMs) synthesize data from multiple data modalities (e.g., image, text), typically building on top of a large language model (LLM) for natural language understanding. Many state-of-the-art open-source LMMs (e.g., LLaVA [ 4 , 20 ] , Molmo [ 5 ] ) leverage a trained CLIP image encoder to compute visual features \phi(x_{i}) from input image x_{i} . A vision-language connector h:\mathcal{Z}_{v}\rightarrow\mathcal{Z}^{\prime}_{t} is trained to map