Improving Robustness of Foundation Models in Domain Adaptation with Soup-Adapters

Marco Roschkowski · 2025

Read Improving Robustness of Foundation Models in Domain Adaptation with Soup-Adapters by Marco Roschkowski (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05807.txt.

Introduction

Marco Roschkowski published this machine learning paper in 2025. It tackles few-shot domain adaptation, where a large pre-trained foundation model must adjust to a new task from only a few examples. Two problems stand in the way. Tuning the settings is hard because there is little spare data to test them on. Also, small shifts between training and test data can break the model.

The proposed fix, called Soup-Adapters, trains several small adapter modules independently, each with different settings, and then averages what they produce. The blend beat every single adapter and held up better when the data shifted. It was also far less touchy about one key setting of the CLIP-Adapter method. Because the blend can be folded back into a single adapter, the final model stays compact. The study also tests the idea on a second vision model, DINOv2.

Catalog record

CREATORMarco Roschkowski
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05807

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Robustness of Foundation Models in Domain Adaptation with Soup-Adapters Marco Roschkowski Abstract In this paper, we tackle two fundamental problems in few-shot domain adaptation of foundation models. First, hyperparameter tuning is often impractical due to the lack of large validation datasets. Second, model robustness under distribution shifts where test time data deviates slightly from training distributions, remains a concern. We show that by training multiple independent adapters and averaging their outputs, the new model has a higher performance and is more robust to distribution shifts compared to any individual adapter. This improvement holds even when the adapters are trained with diverse hyperparameters sampled from a wide range, resulting in varied individual performance. Consequently, our method addresses both of the problems described above. The ensemble is also significantly less sensitive to the residual ratio, a critical hyperparameter of CLIP-Adapter. Since the ensemble can be reparameterized to a single adapter again using a principled concatenation of the parameters, we refer to our method as Soup-Adapter. This is also the first study to explore CLIP adapter-style techniques for DINOv2 and to directly compare them with CLIP in this setting. keywords CLIP, DINOv2, CLIP-Adapter, multi-modal learning, self-supervised learning, foundation models 1 Introduction Computer vision has seen tremendous progress due to the emergence of deep learning technologies. Large supervised benchmark datasets such as ImageNet ( Deng et al. 2009 ) have led to several methodological breakthroughs. These include overcoming traditional computer vision methods in ( Krizhevsky et al. 2012 ) , the introduction of skip connections in ( He et al. 2016 ) , advanced architectures such as inverted bottlenecks in ( Sandler et al. 2018 ) and improved scaling techniques in ( Koonce and Koonce 2021 ) . A long-standing limitation has been the dependence on such large curated datasets which are expensive to obtain. Recently, the paradigm of foundation models has become an attractive alternative in which a single model is being trained on a corpus of data large enough to generalize well on several distinct downstream tasks. One notable vision foundation model is CLIP ( Radford et al. 2021 ) which learns a joint embedding space of images and their corresponding captions. This architecture naturally has the ability to perform zero-shot classification by describing visual categories via text prompts. Another popular foundation model is DINOv2 ( Oquab et al. 2023 ) which has been trained on a large curated corpus of images to produce robust features. These models can be easily adapted for few-shot learning using KNN evaluation or prototypical learning ( Snell et al. 2017 ) . However, CLIP models do not achieve the same zero-shot performance as state-of-the-art supervised models. Similarly, DINOv2 models do not achieve the same KNN-evaluation accuracy as comparable fully supervised models. This has led to a growing interest in few-shot adaptation techniques where the goal is to boost the performance of foundation models on the specific downstream task using a small number of additional labeled samples. In CoOp ( Zhou et al. 2022b ) , it has been shown that the learning of optimal text prompts can improve downstream performance. Adapters such as the CLIP-Adapter ( Gao et al. 2024 ) have become a dominant line of research where the foundation model itself is kept frozen and only additional parameters are being learned. CLIP adapters typically have a very low number of parameters compared to the foundation model itself. This work aims to address two challenges that are common in few-shot learning with adapters. The first problem is that it is often not possible to perform extensive hyperparameter tuning because large validation datasets are typically not available. A particularly important hyperparameter, called the residual ratio, is often necessary if CLIP-Adapters overfit. After training the adapter, this balancing factor allows one to smoothly interpolate between the original zero-shot features and the trained adapter features. Secondly, our goal is to improve performance under distribution shifts of adapter networks. Our contributions may be summarized as follows. • We propose Soup-Adapter, a simple yet effective ensemble of independently trained CLIP-Adapters, which averages their predictions to improve performance. • We demonstrate that Soup-Adapter not only enhances the accuracy of CLIP-Adapter but also significantly improves robustness under distribution shifts, an important limitation of many domain adaptation methods. • Since Soup-Adapter works well when the individual hyperparameters are sampled randomly from a diverse range of configurations for each component, we effectively tackle the problem of hyperparameter tuning. • The proposed method is less sensitive to the residual ratio. • We demonstrate that CLIP-Adapter-based methods, especially the proposed Soup-Adapter, perform effectively with DINOv2 models. • We show that Soup-Adapter can be reparameterized into a single adapter—sharing the same architecture as the individual adapters in the ensemble—through parameter concatenation. The structure of this work is as follows. We will discuss related work in Section 2 . Our method will be introduced in Section 3 . We thoroughly evaluate our method on several datasets in Section 4 and conclude the paper with Section 5 . 2 Related Work 2.1 Emergent Properties of Foundation Models GPT-3 ( Brown et al. 2020 ) demonstrated that large language models trained on massive corpora have impressive few-shot capacities and generalize well on very different language tasks. In DINO ( Caron et al. 2021 ) , it has been shown that self-supervised vision transformers ( Dosovitskiy 2020 ) can perform zero-shot segmentation and that the embeddings of such models yield high KNN-classification accuracies. These have been improved by better training strategies and a large curated pretraining dataset in ( Oquab et al. 2023 ) . CLIP ( Radford et al. 2021 ) demonstrated that the connection of images and text via an image and a text encoder, respectively, leads to high zero-shot accuracies. Moreover, ImageBind ( Girdhar et al. 2023 ) demonstrated that robust joint embeddings can be achieved across different modalities without direct pairing, instead linking each modality to images. 2.2 Prompting The systematic discovery of better prompts originated in the natural language modeling community; see, for example, ( Jiang et al. 2020 ; Shin et al. 2020 ; Gao et al. 2020 ; Beyer et al. 2020 ; Liu et al. 2024 ; Lester et al. 2021 ) . Prompting has been successively applied to CLIP in CoOP ( Zhou et al. 2022b ) and CoCoO ( Zhou et al. 2022a ) . Specifically, CoOp initializes the text prompts as a set of learnable embeddings, which is then continuously updated by forwarding it through the text encoder and applying gradient descent. This procedure is then refined in CoCoO to by learning prompts while fixing important words that highlight information about the class. Discovering prompts for multi modal models remains an active area of research ( Zang et al. 2022 ; Du et al. 2022 ; Khattak et al. 2023 ; Shen et al. 2024 ) . 2.3 Adapter Techniques Adapter techniques, which freeze the original model weights and train only additional adapter layers, have gained popularity. Their main advantages are improved computational efficiency, since the backbone is not updated, and greater robustness to distribution shifts in few-shot settings. CLIP-Adapter ( Gao et al. 2024 ) introduced residual MLP layers to incorporate new knowledge from few-shot data into the

Read the complete text