Enhancing CLIP Conceptual Embedding through Knowledge Distillation

Kuei-Chun Kao · 2024

Read Enhancing CLIP Conceptual Embedding through Knowledge Distillation by Kuei-Chun Kao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03513.txt.

Introduction

Kuei-Chun Kao teaches a famous image-text model new depth by distilling knowledge from a large language model into it. CLIP aligns pictures and captions, but its encoders skim shallow features from both. His 2024 paper presents Knowledge-CLIP, trained with a distillation scheme based on Llama 2. Three objectives work together: the text encoder is trained to mirror the teacher's richer embeddings; concept learning assigns soft concept labels; and contrastive learning keeps the image-text alignment sharp.

A teacher that has read the world can lend its concepts to a model that only saw captions. Soft concepts, graded rather than binary, carry nuance a label list cannot. The author works solo on a widely used foundation model. Gains here ripple into every search, tagging, and zero-shot tool built on CLIP. The paper is a recipe for upgrading a standard part.

Catalog record

CREATORKuei-Chun Kao
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03513

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing CLIP Conceptual Embedding through Knowledge Distillation Kuei-Chun Kao Abstract Recently, CLIP has become an important model for aligning images and text in multi-modal contexts. However, researchers have identified limitations in the ability of CLIP’s text and image encoders to extract detailed knowledge from pairs of captions and images. In response, this paper presents Knowledge-CLIP, an innovative approach designed to improve CLIP’s performance by integrating a new knowledge distillation (KD) method based on Llama 2. Our approach focuses on three key objectives: Text Embedding Distillation, Concept Learning, and Contrastive Learning. First, Text Embedding Distillation involves training the Knowledge-CLIP text encoder to mirror the teacher model, Llama 2. Next, Concept Learning assigns a soft concept label to each caption-image pair by employing offline K-means clustering on text data from Llama 2, enabling Knowledge-CLIP to learn from these soft concept labels. Lastly, Contrastive Learning aligns the text and image embeddings. Our experimental findings show that the proposed model improves the performance of both text and image encoders. 1 Introduction The pre-training of the multimodal encoders associated with vision language, a good example being CLIP Radford et al. 2021 , has been found to be very helpful in learning transferrable features derived from paired data of image and text. The CLIP learning framework is contrastive, typically relying on data augmentation in order to eliminate unnecessary insertion and shortcuts. However, the impressive results demonstrated by Vision and Language models (VLMs) Gokhale et al. 2023 ; Thrush et al. 2022 ; Yuksekgonul et al. 2023 on myriad recognized benchmarks do not necessarily indicate a comprehensive understanding of the compositional elements of text or images. These models, as exemplified by CLIP, raise questions about their ability to differentiate between sentence structures like "an orangutan eating and an officer flying" and "an orangutan and an officer eating an orangutan" Scenes in nature pose a significant challenge due to their complexity, resulting from the numerous objects and attributions they carry and their mutual interactions. The challenge inherent in CLIP, stemming from its difficulty in addressing image segmentation and object detection tasks due to the need for per-pixel label knowledge, has been addressed by Li et al. 2022 through the introduction of GLIP. The proposed method unified object detection and phrase grounding through pre-training, effectively leveraging external knowledge, i.e., grounding boxes. The integration of such external information facilitates the alignment of image-language data, enhancing the model’s capability in handling complex visual tasks. Inspired by their work, our objective is to integrate external knowledge from established large language models into CLIP, with the aim of further elevating its overall quality. Therefore, we introduce a pioneering methodology named Knowledge-CLIP, comprising triple sets of objectives. Firstly, our focus is on knowledge distillation (KD) from large language models, exemplified by Llama 2 Touvron et al. 2023 , with the aim of enhancing the quality of CLIP’s text encoder. Secondly, we posit that the embeddings generated by Llama 2 encompass more valuable attributes and conceptual information, such as color and action, than CLIP’s text encoder. Thus, employing K-means clustering Hartigan and Wong 1979 on Llama 2’s embeddings, we derive soft concept labels for caption-image pairs. Subsequently, we leverage these soft concept labels to refine the quality of both CLIP’s text and image encoders. Finally, we reused the contrastive objective from CLIP to continually align the text and image embeddings. Figure 1: Overview of our proposed Knowledge-CLIP, which has five modules: CLIP text encoder ( E_{T} ), CLIP image encoder ( E_{I} ), Classifier ( C ), Linear projector ( W_{e} ) and Llama 2 ( L ). 2 Related Works 2.1 Knowledge Distillation Knowledge distillation (KD) is a training technique wherein a neural network, referred to as a student, is trained to replicate parts of another neural network, known as a teacher Ba and Caruana 2014 . The most common approach involves matching the output of the teacher network. However, an alternative option is to match the hidden layers, providing a more nuanced transfer of knowledge Romero et al. 2015 ; Aguilar et al. 2020 . In terms of the loss functions utilized during this process, KL divergence is a common choice for matching probability outputs, while L2 norm is frequently employed to align the hidden vectors Kim et al. 2021 . This technique allows for the compact representation of knowledge learned by the teacher network to be transferred to the student network, enhancing the student’s performance and generalization capabilities. 2.2 CLIP’s Text Encoder In the context of multimodal models, the text-to-image query such as "A UCLA CS student in a futuristic lab, donned in virtual reality gear and programming a robotic assistant next to a professor in a lab coat." exemplifies the expectations for modern multimodal models. This type of query demands spatial precision (e.g., specifying the position of entities), compositional understanding (highlighting certain attributes like a UCLA CS student but not a UCLA CS assistant), and a touch of imagination, describing scenarios that may not exist in reality. However, recent works Gokhale et al. 2023 ; Thrush et al. 2022 ; Yuksekgonul et al. 2023 shed light on a notable challenge. Despite achieving robust benchmark performance, various multimodal models often struggle with even basic reasoning tasks, particularly those involving spatial relations or attribute attachments. These findings underscore the existing limitations in the reasoning capabilities of multimodal models, especially when confronted with intricate and imaginative textual input. 3 Method 3.1 Problem Definition and Annotations For the sake of completeness, we first define the setting and notations considered in this paper. During training, we have N caption-image pairs denoted as X=\{X_{1},X_{2},...,X_{N}\} . For the i th caption-image pair, we have a caption and an image, i.e., X_{i}=(x^{C}_{i},x^{I}_{i}) . As shown in Figure 1 , our proposed Knowledge-CLIP has five modules: CLIP text encoder ( E_{T} ), CLIP image encoder ( E_{I} ), Classifier ( C ), Linear projector ( W_{e} ) and Llama 2 ( L ). 3.2 Text Embedding Distillation In a previous KD work Jiao et al. 2019 , they effectively distill the knowledge from BERT base to TinyBERT with mean squared error loss function (MSE). Thus we distill the output embeddings from Llama 2, and the objective is as follows: \text{\normalsize}L_{emb}=\sum_{i=1}^{N}\|(E_{T}(x_{i}^{C}))W_{e}-L(x_{i}^{C})\|_{2}^{2}, (1) where the matrix W_{e}\in\mathbf{R}^{d\times d^{\prime}} . The scalar values d and d^{\prime} denote the hidden sizes of the outputs from CLIP text encoder ( E_{T} ) and Llama 2 ( L ). Noted that the matrix W_{e} is a learnable linear transformation, which transforms the hidden states of E_{T} into the same space as the L . 3.3 Concept Learning Given a caption, there exist multiple attributes or concepts, e.g., color, position, action, etc. However, CLIP text encoder failed to extract these information observed from previous papers. We hypothesize that captions having the same attributes and concepts would have similar embeddings from Llama 2. Therefore, we utilize k-means clustering method to categorize the output embeddings from Llama 2 with inputting captions. Then, we regard the result from k-means clustering as the soft concept labels, denoted as S=\{s_{1},s_{2},...,s_{N}\} . Noted that s_{i}\in[K] is a K-class categorical variable. After obtaining soft concept labels, we use the Llama 2’s embeddings and their corresponding soft labels to train the Classifier ( C ).

Read the complete text