Assessing and Learning Alignment of Unimodal Vision and Language Models

Le Zhang, Qian Yang, Aishwarya Agrawal · 2024

Read Assessing and Learning Alignment of Unimodal Vision and Language Models by Le Zhang, Qian Yang, Aishwarya Agrawal (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04616.txt.

Introduction

Le Zhang, Qian Yang, and Aishwarya Agrawal test how well lone vision and language models already line up. Their 2024 paper probes alignment with a direct, probe-style method closer to real use than past checks. Self-taught vision models aligned by their training aim, they find, and cluster quality in the features beat plain separability as the driver.

Their frame, SAIL, then aligns pretrained single-mode models with a sliver of the paired data a from-scratch rival needs. One card and hours of training sufficed. Scores edged past that rival on a flagship test and shone in retrieval, reasoning, and segmentation.

Catalog record

CREATORLe Zhang, Qian Yang, Aishwarya Agrawal
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04616

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Assessing and Learning Alignment of Unimodal Vision and Language Models Le Zhang, Qian Yang, Aishwarya Agrawal Abstract How well are unimodal vision and language models aligned? Although prior work have approached answering this question, their assessment methods do not directly translate to how these models are used in practical vision-language tasks. In this paper, we propose a direct assessment method, inspired by linear probing, to assess vision-language alignment. We identify that the degree of alignment of the SSL vision models depends on their SSL training objective, and we find that the clustering quality of SSL representations has a stronger impact on alignment performance than their linear separability. Next, we introduce Swift Alignment of Image and Language (SAIL), a efficient transfer learning framework that aligns pretrained unimodal vision and language models for downstream vision-language tasks. Since SAIL leverages the strengths of pretrained unimodal models, it requires significantly fewer ( ∼ \sim 6%) paired image-text data for the multimodal alignment compared to models like CLIP which are trained from scratch. SAIL training only requires a single A100 GPU, ∼ \sim 5 hours of training and can accommodate a batch size up to 32,768. SAIL achieves 73.4% zero-shot accuracy on ImageNet (vs. CLIP’s 72.7%) and excels in zero-shot retrieval, complex reasoning, and semantic segmentation. Additionally, SAIL improves the language-compatibility of vision encoders that in turn enhance the performance of multimodal large language models. 1 1 1 Project page: https://lezhang7.github.io/sail.github.io/ 1 Introduction Figure 1 : Conceptual Overview : Alignment probing evaluates the alignment potential of two pretrained uni-modal models. The integration of language and vision is pivotal in advancing models for computer vision tasks [ 27 , 28 , 19 , 20 , 46 ] . This raises a key question: “To what extent can unimodal visual and language models be aligned with each other?” In particular, how do unimodal representations impact cross-modal alignment: Do larger unimodal models trained on extensive datasets yield better cross-modal alignment? Does the choice of self-supervised learning (SSL) method play a critical role in determining the alignment strength? What property of SSL representation correlates the most with cross-modal alignment performance: is it linear separability or the clustering quality? Although prior work have approached answering the question of “To what extent are pretrained unimodal models aligned with each other” , their assessment methods do not directly translate to how these models are used in practical vision-language tasks. For instance, Huh et al. [17] assess cross-modal alignment using mutual nearest-neighbor metrics, indicating a certain level of alignment across models trained on separate modalities. However, this assessment method serves only as a proxy method as it focuses on relative ordering within each modality rather than directly measuring cross-modal distances. The latter is precisely how inference with vision-language models (VLMs) is conducted, so we believe directly measuring cross-modal distances is a more direct measurement of alignment performance. To quantitatively answer aforementioned questions regarding the impact of modality-specific representation on cross-modal alignment, we introduce visual-language alignment probing , akin to linear probing used in SSL evaluation. As illustrated in Fig. 1 , alignment probing freezes the pretrained vision and language backbones and trains a lightweight linear alignment layer on image-text datasets. We evaluate alignment through zero-shot retrieval tasks, finding that vision and language models exhibit strong alignment generally. However, the degree of alignment of the SSL vision models depends on their SSL training objective. We also found that the clustering quality of visual representations, as indicated by k-NN classifier performance, has a stronger impact on image-text alignment performance than their linear separability. Furthermore, for complex visio-linguistic reasoning, strong language understanding is essential. CLIP training even with scaled model and training data size is insufficient for developing a high-quality text encoder. Building on the findings that unimodal vision and language models show inherent alignment, and that language models trained on extensive natural language data serve as effective text encoders for VLMs, we propose Swift Alignment of Image and Language (SAIL) for learning better vision-language alignment. SAIL is an efficient transfer learning framework designed to construct robust foundational VLMs leveraging high-quality pretrained unimodal vision and language models. To enhance alignment quality, we employ three optimized components: a non-linear alignment layer, a refined contrastive loss function, and MLLM-generated high-quality training captions. SAIL is highly data-efficient, leveraging pretrained unimodal models and requiring only about 6% of the paired image-text data needed for models like CLIP, which are trained from scratch. It’s also compute-efficient, needing only a single A100 GPU, \sim 5 hours of training, and supporting batch sizes up to 32,768 by training just the alignment layer. By aligning the pretrained vision-encoder DINOv2-L and the pretrained language encoder NV2 using 23M image-text pairs, our method SAIL outperforms CLIP trained on 400M image-text pairs by 0.7% on ImageNet, and by 5.6% and 2.7% on COCO text-to-image and image-to-text retrieval respectively. SAIL leverages the strengths of unimodal models – DINOv2’s fine-grained visual understanding and NV2’s complex language reasoning – and excels considerably in challenging vision-language tasks such as Winoground [ 40 ] and MMVP [ 42 ] , as well as in open-vocabulary image segmentation tasks. Furthermore, in comparison with prior efficient training methods like LiT [ 44 ] and Sharelock [ 36 ] , which focus on tuning language models to align with frozen vision encoders, SAIL improves both the alignment performance as well as the language-compatibility of the vision encoder itself. This enables SAIL’s vision encoder to be transferable to MLLMs, resulting in significant performance gains; when integrated with LLaVA-1.5 [ 27 ] , SAIL’s alignment training pushes the capabilities of the DINOv2 vision encoder from lagging behind the CLIP vision encoder to surpassing it in 5 out of 7 tasks downstream MLLM tasks. Figure 2 : Linear alignment probing results trained with 2.2M paired data from CC3M. The radius represents the relative number of parameters in each model. The Y-axis indicates the zero-shot MSCOCO retrieval average R@10 performance. (Left) the X-axis shows kNN performance for various SSL models. (Right) the X-axis displays MTEB average scores across models. 2 Part I: Assessing Alignment between Unimodal Models 2.1 Approach and Experimental Setup In this section, we evaluate the alignment potential of pretrained unimodal models to determine those most compatible with models of the other modality. To focus on the alignment capacity of the pretrained models, we use a linear alignment layer to connect their representations. We refer to this as alignment probing . We use linear layers instead of MLPs because the latter could introduce additional alignment capability and hence confound the findings. The alignment probing architecture is illustrated in Fig. 1 : only the alignment layer is trained, while the backbones remain frozen. We employ contrastive learning to pull matched image-text pairs closer and push unmatched pairs further in the representation space; see Appendix for training details. We use the open-source CC3M dataset (2.2M paired image-text samples) to train the alignment layer, leveraging its diversity and quality as an effective probing dataset. To measure the alignment quality, we test on COCO in zero-shot retrieval setup, using the R@10 metric. We report average recall of text-to-image and image-to-text retrieval tasks. For systematic evaluation, we fix an anchor model in one modality and vary models in the other modality to

Read the complete text