Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models

Sri Harsha Dumpala, David Arps, Sageev Oore, Laura Kallmeyer, Hassan Sajjad · 2024

Read Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models by Sri Harsha Dumpala, David Arps, Sageev Oore, Laura Kallmeyer, Hassan Sajjad (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08111.txt.

Introduction

Sri Harsha Dumpala and co-authors probe how well vision-language models grasp sentence structure. These models pair a text encoder with image tools and power tasks like caption writing. Past work found weak spots in meaning, but the cause was unclear. The team tests how much syntax, the rules that order words, each encoder holds. They compare joint models with text-only models of varied size and training.

Text-only encoders picked up syntax better than the joint models did. What shaped syntax most was the training goal, not model size or data volume. One model, CLIP, lost syntax in deeper layers, while others held it best in middle layers. The results help explain why joint models can stumble on word order.

Catalog record

CREATORSri Harsha Dumpala, David Arps, Sageev Oore, Laura Kallmeyer, Hassan Sajjad
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08111

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models Sri Harsha Dumpala, David Arps, Sageev Oore, Laura Kallmeyer, Hassan Sajjad Abstract Vision-language models (VLMs), serve as foundation models for multi-modal applications such as image captioning and text-to-image generation. Recent studies have highlighted limitations in VLM text encoders, particularly in areas like compositionality and semantic understanding, though the underlying reasons for these limitations remain unclear. In this work, we aim to address this gap by analyzing the syntactic information, one of the fundamental linguistic properties, encoded by the text encoders of VLMs. We perform a thorough analysis comparing VLMs with different objective functions, parameter size and training data size, and with uni-modal language models (ULMs) in their ability to encode syntactic knowledge. Our findings suggest that ULM text encoders acquire syntactic information more effectively than those in VLMs. The syntactic information learned by VLM text encoders is shaped primarily by the pre-training objective, which plays a more crucial role than other factors such as model architecture, model size, or the volume of pre-training data. Models exhibit different layer-wise trends where CLIP performance dropped across layers while for other models, middle layers are rich in encoding syntactic knowledge. 1 Introduction Vision-Language Models (VLMs), such as CLIP, have demonstrated remarkable performance across a range of downstream tasks, including image-text retrieval, image captioning, and text-to-image generation [ 31 , 24 , 25 , 32 , 35 ] . However, it remains unclear which linguistic features are encoded by the text encoders of VLMs to achieve this level of performance. Recent studies have demonstrated that Unimodal Language Models (ULMs), such as BERT [ 7 ] and RoBERTa [ 27 ] , encode linguistic information, including semantic and syntactic sentence structure [ 18 , 28 ] . Do the text encoders of VLMs (TE-VLM) also exhibit this ability to encode linguistic information, as ULMs do? In this direction, recent works have focused on analyzing the compositional abilities of TE-VLM. These works have shown that TE-VLMs struggle to represent (encode) the word order in the input text; instead, they tend to represent the input text in a bag-of-words format [ 40 , 46 , 33 , 45 ] . Sentence structure is important to correctly understand the meaning conveyed in an input sequence and is crucial for tasks such as image-text matching or image generation. Figure 7 illustrates that even recent large VLMs often lack the understanding of syntactic relations in a sentence. The sentence structure of the prompt, “A cat chases a dog” , contains a clear description of the roles of the cat and the dog. This is expressed through the grammatical structure of the sentence in general, and through the grammatical roles nsubj (nominal subject) and obj (direct object) of the animals in particular. However, StableDiffusion 3.5 fails to consistently generate images in which the cat is the chaser (the nsubj of the sentence). This means that either the text encoder does not appropriately encode the prompt structure, or the information is encoded but the image decoder fails to use it. In this study, we investigate the first of these possible causes. A cat chases a dog det nsubj root det dobj Figure 1 : Four images sampled from the StableDiffusion 3.5 model for the same prompt A cat chases a dog , and the Universal Dependency tree for the prompt. StableDiffusion 3.5 uses CLIP-L, CLIP-bigG and T5-XXl models to encode the input prompt. Accessed via https://huggingface.co/spaces/stabilityai/stable-diffusion-3.5-large-turbo . We experimented with variations of this prompt and various other prompts, with similar issues observed in the results. While previous works have analyzed the distribution of linguistic information (both syntactic and semantic) across layers of ULMs [ 17 , 13 , 20 , 22 , 14 ] , no studies have comprehensively evaluated the presence of and the distribution of syntactic information in TE-VLMs. It is not yet clear how syntactic information is distributed across the layers of TE-VLMs or whether TE-VLMs exhibit similar patterns to ULMs. A major difference between VLMs and ULMs stems from the objective function used to pre-train these models. Most ULMs [ 8 , 27 ] are pre-trained using token-level objectives, i.e., predicting masked tokens, or predicting next tokens. Most VLMs, however, are trained using contrastive learning scheme to embed both images and text into a shared representation space. In this study, we analyze the distribution of syntactic information across the network of TE-VLMs and compare it with that of ULMs. To understand the impact of the pre-training objective function on linguistic information encoded by the VLMs, we analyze a variety of VLMs pre-trained with different objective functions in addition to contrastive loss objective function. To this end, our study aims to answer the main research question of “Do VLMs encode syntactic knowledge?” We divide it into the following sub-questions: 1. How do the syntactic learning capabilities of TE-VLMs compare to those of ULMs? 2. What is the influence of pre-training objective function in a VLM’s ability to learn syntactic knowledge? 3. How does the syntactic information vary across the layers of TE-VLMs? We use DepProbe [ 30 ] to probe representations of VLMs in their ability to predict Universal Dependencies [ 6 ] . We perform experiments using various variants of CLIP [ 31 ] and FLAVA [ 38 ] and compare the results with several ULMs. Our notable findings are summarized below: • TE-VLMs perform poorly in learning syntax when compared to ULMs. • Syntactic knowledge in representation is strongly influenced by the pre-training objective. • Models exhibit different layer-wise trends where CLIP performance dropped across layers while other VLMs and ULMs show higher syntactic knowledge at middle layers. • Using contrastive loss for training, increasing the number of parameters and the training data size do not improve the syntactic learning of a model. • CLIP struggles in learning relations between content words such as predicate-argument relations. However, functional relations such as determiners are predicted well by all models. 2 Related work 2.1 Vision-language models (VLMs) Recent work has introduced several VLMs designed to learn the multimodal interactions between vision and language data. These models vary in architecture, size, pre-training objectives, and the amount of pre-training data [ 31 , 36 , 38 ] . Radford et al. [31] introduced CLIP (Contrastive Language–Image Pre-training), which employs a contrastive learning framework to embed both images and text into a shared representation space. CLIP demonstrated impressive performance across several tasks including zero-shot image classification. Subsequent improvements to CLIP include increasing the model parameters and training dataset size [ 36 ] , training with large number of noisy image-caption pairs [ 15 ] , and incorporating additional objective functions beyond contrastive loss, such as masked language modeling, masked image modeling, and image-text matching [ 38 , 23 , 24 ] . The effects of these developments on the learning capabilities of VLMs, particularly regarding syntax understanding, have not been thoroughly explored in previous studies. In this work, we analyze the impact of these modifications on the syntactic understanding abilities of VLMs. 2.2 Sentence Structure in VLM text encoders Recent studies have focused on evaluating the linguistic and relational understanding abilities of VLMs [ 45 , 43 , 21 , 41 ] .

Read the complete text