Active Learning via Vision-Language Model Adaptation with Open Data
Tong Wang, Jiaqi Wang, Shu Kong · 2025
Read Active Learning via Vision-Language Model Adaptation with Open Data by Tong Wang, Jiaqi Wang, Shu Kong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01724.txt.
Introduction
Tong Wang, Jiaqi Wang, and Shu Kong wrote a 2025 computer science paper on teaching vision-language models with less hand-labeled data. Such models learn from huge web data first, then need labeled examples for each new task. Labeling costs money and often takes expert skill.
Active learning cuts that cost by picking only the most useful examples to label. The team's twist is to also pull in free open data that matches the task. They call the method Active Learning with Open Resources. Adding these found examples raised scores by a lot in their tests. They also compare ways to tune the model once the data is picked.
Catalog record
| CREATOR | Tong Wang, Jiaqi Wang, Shu Kong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Active Learning via Vision-Language Model Adaptation with Open Data Tong Wang, Jiaqi Wang, Shu Kong Abstract Pretrained on web-scale open data, Vision-Language Models (VLMs) offer powerful capabilities for solving downstream tasks, after being adapted to task-specific labeled data. However, data labeling can be expensive, especially when domain expertise is required. Active Learning (AL) aims to reduce this expense by strategically selecting the most informative data for labeling and model training. Recent AL methods have explored open-source VLMs but have yet to fully leverage publicly available open data, such as datasets used for pretraining VLMs. In this work, we propose leveraging VLM’s pretraining data by retrieving samples closely related to the downstream task, using them to augment the task-specific data for AL. As expected, incorporating this data into existing AL methods leads to significant performance improvements. Given that our method exploits open-source VLM and open data, we refer to it as Active Learning with Open Resources (ALOR) . Additionally, most VLM-based AL methods use prompt tuning (PT) for model adaptation, likely due to its ability to directly utilize pretrained parameters and the assumption that doing so reduces the risk of overfitting to limited labeled data. We rigorously compare popular adaptation approaches, including linear probing (LP), finetuning (FT), and contrastive tuning (CT). Our results reveal two key findings: (1) all adaptation approaches benefit significantly from incorporating retrieved data, and (2) CT consistently outperforms other adaptation approaches regardless what AL methods to apply, even when not using retrieved data. Further analysis of retrieved data reveals a naturally imbalanced distribution of task-relevant classes, exposing inherent biases within the VLM. This insight motivates our novel Tail First Sampling (TFS) strategy for AL, an embarrassingly simple yet effective method that prioritizes sampling data from underrepresented classes to label. Extensive experiments on standard benchmark datasets demonstrate that our final method, contrastively finetuning VLM on both retrieved and TFS-selected labeled data, achieves state-of-the-art performance, significantly surpassing existing methods. Figure 1 : Active Learning with Open Resources (ALOR) exploits open-source VLM and open data (especially VLM’s pretraining data), unlike recent AL methods which use only the former. Specifically, for task-specific class names, ALOR retrieves relevant pretraining data to augment the limited number of task-specific labeled data. The retrieved data not only reveals an imbalanced distribution but also unveils how the VLM is biased and how task-specific unlabeled data is (similarly) imbalanced. Leveraging these insights, ALOR adopts Tail First Sampling (TFS) that prioritizes sampling unlabeled examples data from underrepresented classes to label. ALOR adapts the VLM by finetuning it over both the retrieved and TFS-selected labeled data, achieving significantly better performance than prior arts by + 7% accuracy on multiple benchmarks (Table 2 ). 1 Introduction Pretrained Vision-Language Models (VLMs) [ 39 , 55 ] demonstrate strong zero-shot performance on various visual recognition [ 36 ] . Yet, to excel in a downstream task, a VLM requires adaptation, e.g., through finetuning on labeled task-specific data. However, data labeling is expensive and often demands domain expertise. Pool-based Active Learning (AL) aims to reduce this expense by strategically selecting informative unlabeled data for labeling and model training [ 2 , 1 , 21 , 38 , 46 ] . Status Quo. The core to AL is data selection strategies, which are commonly based on uncertainty [ 11 , 16 , 22 ] , diversity [ 38 , 46 ] , or both [ 17 , 18 , 2 ] . Recent methods have exploited open-source VLMs for data selection and adapted them for AL [ 43 , 3 ] . Yet, it primarily adopts prompt tuning (PT) [ 60 , 20 , 59 ] as an VLM adaptation approach, likely due to its ability to directly leverage the pretrained parameters to reduce the risk of overfitting to the limited labeled data. That said, it is underexplored whether other adaptation approaches such as finetuning can perform better than PT. Furthermore, despite exploiting an open-source VLM, contemporary AL methods have yet to leverage publicly-available open data, e.g., the VLM’s pretraining data [ 31 , 32 ] or Internet data [ 26 ] , but related domains have done so such as few-shot learning. Technical Insights. For AL, in addition to using VLM [ 43 , 3 ] , we embrace open data, particularly the VLM’s open-source pretraining data [ 44 , 45 ] (for reproduction). We retrieve pretraining examples relevant to the downstream task and use them to augment labeled data. Unsurprisingly, adapting the VLM using both retrieved and labeled data greatly enhances AL ( Table 1 ). Moreover, we rigorously compare different adaptation approaches including prompt tuning (PT), linear probing (LP), tuning (FT), and contrastive finetuning (CT). Our results show that CT resoundingly outperforms the others, regardless whether to exploit the retrieved data and what AL methods to use. Importantly, the retrieved data reveals imbalanced distributions of task-relevant classes, implying how the VLM is biased and how the unlabeled task-specific data is similarly imbalanced (as they are sampled in the real world). This insight motivates our proposed Tail First Sampling (TFS) strategy, which is an embarrassingly simple yet effective AL method that prioritizes sampling unlabeled data for underrepresented classes to label. We call our method Active Learning with Open Resources (ALOR) , as it exploits both open-source VLM and open data. We summarize ALOR in Fig. 1 , which consists of three major components: retrieval-based data augmentation (RDA), VLM adaptation, and the TFS strategy. Our ALOR significantly outperforms prior arts ( Table 2 ). Contributions. We make three major contributions: 1. We study AL by embracing both a VLM and its pretraining data (instantiating the open data ). Particularly, we present retrieval-based data augmentation (RDA) to retrieve VLM’s pretraining data relevant to the downstream task, greatly enhancing existing AL methods. 2. We observe that the retrieved data follows imbalanced distributions, implying how the VLM is biased and how unlabeled task-specific data is similarly imbalanced distributed. This motivate our simple yet novel Tail First Sampling (TFS) strategy that prioritizes rare classes in data selection. Extensive experiments show that TFS outperforms prior AL methods. 3. We rigorously compare different VLM adaptation approaches including finetuning (FT), contrastive tuning (CT), linear probing (LP) and prompt tuning (PT). We show that CT significantly outperforms the others, even when retrieved data is not used. Our final method assembles CT, RDA and TFS, achieving the state-of-the-art on five benchmarks. 2 Related Work Active Learning (AL). Meticulous data labeling is crucial to machine-learning solutions for real-world applications, but data labeling is often tedious, laborious, and costly that requires domain expertise [ 48 , 15 , 35 ] . AL mitigates this cost by strategically selecting the most informative unlabeled data to label during model training [ 47 , 40 ] . Recent AL methods usually rely on uncertainty [ 16 ] , diversity [ 41 , 11 , 52 ] , or both [ 2 ] . Uncertainty-based methods select unlabeled examples which have high uncertainties of model’s predictions, e.g., high entropy [ 16 ] . Such methods concentrate on forecasting the likelihood of classes while ignoring the distribution of the unlabeled data [ 41 , 11 , 52 ] . In contrast, diversity-based methods consider the distribution of the unlabeled