Attention Head Purification: A New Perspective to Harness CLIP for Domain Generalization

Yingfan Wang, Guoliang Kang · 2024

Read Attention Head Purification: A New Perspective to Harness CLIP for Domain Generalization by Yingfan Wang, Guoliang Kang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07226.txt.

Introduction

Yingfan Wang and Guoliang Kang rethink how to use a big vision-language model across new domains in a 2024 paper. The model, CLIP, matches images with text and starts strong on fresh tasks. But its store of knowledge also holds domain-specific cues that can mislead it elsewhere.

The authors sort the model's attention heads instead of tuning the whole net. Some heads carry task sense; others carry domain habits. Light adapters tune each head for the task. A gate then picks heads whose features stay steady across domains. Training runs both steps at once, with a loss that rewards steady features. Tests on standard domain-shift benchmarks showed clear gains, though the method stays simple.

Catalog record

CREATORYingfan Wang, Guoliang Kang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-07226

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention Head Purification: A New Perspective to Harness CLIP for Domain Generalization Yingfan Wang, Guoliang Kang Abstract Domain Generalization (DG) aims to learn a model from multiple source domains to achieve satisfactory performance on unseen target domains. Recent works introduce CLIP to DG tasks due to its superior image-text alignment and zeros-shot performance. Previous methods either utilize full fine-tuning or prompt-learning paradigms to harness CLIP for DG tasks. Those works focus on avoiding catastrophic forgetting of the original knowledge encoded in CLIP but ignore that the knowledge encoded in CLIP in nature may contain domain-specific cues that constrain its domain generalization performance. In this paper, we propose a new perspective to harness CLIP for DG, i.e., attention head purification. We observe that different attention heads may encode different properties of an image and selecting heads appropriately may yield remarkable performance improvement across domains. Based on such observations, we purify the attention heads of CLIP from two levels, including task-level purification and domain-level purification . For task-level purification, we design head-aware LoRA to make each head more adapted to the task we considered. For domain-level purification, we perform head selection via a simple gating strategy. We utilize MMD loss to encourage masked head features to be more domain-invariant to emphasize more generalizable properties/heads. During training, we jointly perform task-level purification and domain-level purification. We conduct experiments on various representative DG benchmarks. Though simple, extensive experiments demonstrate that our method performs favorably against previous state-of-the-arts. 1 Introduction Deep learning has attained remarkable success on various downstream tasks in computer vision, typically under the assumption that both training and test samples are identically distributed. However, in practice, test data distributions (target) are usually different from the training ones (source). In such cases, the performance of deep neural networks may degenerate severely on target, showing a poor domain generalization ability. To mitigate the domain shift, a series of Domain Generalization (DG) methods [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] are proposed to transfer the knowledge learned from multiple source domains to unseen target domains, via domain-invariant learning [ 8 , 9 , 10 , 11 , 12 , 13 ] , meta-learning techniques [ 14 , 15 , 16 , 5 ] , or specifically-designed data augmentations [ 17 , 18 , 19 ] . The remarkable progress achieved by those works can be largely attributed to a well-initialized feature extractor provided by ImageNet-pretrained [ 20 ] backbones. Recent proposed Contrastive Language–Image Pre-training ( i.e., CLIP [ 21 ] ) learns from large amounts of image-text pairs [ 22 ] and demonstrates impressive zero-shot learning performance. It may potentially serve as a better foundation for mitigating the performance gap across domains for specific downstream tasks. Despite superior zero-shot classification performance of CLIP, it is not trivial to harness CLIP for specific domain generalization tasks beyond its zero-shot classification ability. Directly fine-tuning CLIP may obtain even worse performance than zero-shot classification [ 23 , 24 ] . Previous works tackle this issue mainly by introducing various regularizations to avoid the forgetting of original knowledge encoded in CLIP. For example, [ 25 ] fully fine-tunes CLIP’s image encoder with modified contrastive loss to mitigate overfitting and proposes beta moving average to perform a temporal ensembling along the fine-tuning trajectory. [ 2 ] proposes mutual information regularization between the original CLIP and the fine-tuned one to prevent the two models from deviating too much. Although those works achieve improved domain generalization performance, a natural question arises: is the way to avoid knowledge forgetting sufficient to harness CLIP for domain generalization? Figure 1: Left : We use different strategies to evaluate the importance of each attention head on domain generalization and drop the least important ones to see how accuracy changes. The strategies we adopt include “randomly drop( – \lozenge – )”, “manually drop( – \CIRCLE – )”, “drop by cross-validation( – \blacktriangle – )”, and “adapt & drop( – \blacksquare – )”. Details of strategies can be found in Appendix. Via appropriately dropping heads, CLIP’s domain generalization performance can be improved. Right: Attention map [ 26 ] generated by specific heads. The middle columns are from least important heads determined by the cross-validation strategy. They all capture lots of background information. The last column represents the most important one whose attention map mainly focuses on the object itself. The experiments are conducted on OfficeHome [ 27 ] . Best viewed in color. Recent work [ 28 ] shows that different attention heads of CLIP’s image encoder (ViT-based) may encode different properties of an image. Inspired by this work, we conduct ablation experiments to verify the effectiveness of each head in the context of domain generalization. Specifically, we utilize different ways to evaluate the importance of attention heads on domain generalization ( e.g., manual evaluation, evaluation by cross-validation, etc. ) and drop the least important ones. As shown in Figure 1 (a), we observe that via appropriately dropping some attention heads, we may achieve a much better domain generalization performance. This phenomenon implies that not all attention heads are domain generalizable and some heads may harm the domain generalization ability of CLIP, which means those heads may contain non-generalizable cues ( i.e., domain-specific cues) . To further illustrate our findings, we provide attention visualizations in Figure 1 (b). We observe that the attention maps [ 26 ] of the least important heads mainly highlight task-irrelevant regions like background, while the most important head pays more attention to the object itself. As a result, simply avoiding the knowledge forgetting of CLIP may not be optimal for domain generalization tasks. We need to purify the attention heads to make CLIP more task-adapted and domain-generalizable. In this paper, we propose a new perspective to harness CLIP for domain generalization tasks, i.e., through attention head purification. Specifically, we perform two kinds of attention head purification during training, which are named task-level purification and domain-level purification . For task-level purification, we aim to purify the attention head of CLIP to maintain more task-related knowledge. Technically, we adopt LoRA to realize this goal. Different from conventional LoRA implementation, we design head-aware LoRA (HA-LoRA) to purify and adapt each head more accordingly. For domain-level purification, we aim to purify the attention heads of CLIP to make the resulting features more invariant across domains. We design a simple learnable gating strategy to select the heads that most benefit the domain generalization performance. To realize this, in addition to the cross-entropy loss on source images, we utilize Maximize Mean Discrepancy (MMD) loss to encourage the gates to emphasize more domain-invariant head features. Note that we do not utilize the gradients of MMD loss to update the HA-LoRA. In this way, we may decouple task-level and domain-level purification to an extent, i.e., we expect HA-LoRA to focus only on encoding rich task-related properties, while leaving the goal of selecting domain-invariant properties to domain-level purification. During training, we jointly train the head-aware LoRA and the gates of attention heads to perform the task and domain-level purification simultaneously. Pre-trained parameters in the image encoder and the text encoder of CLIP are frozen throughout the training. We conduct extensive experiments on various representative

Read the complete text