ADROIT: A Self-Supervised Framework for Learning Robust Representations for Active Learning
Soumya Banerjee [email protected], Vinay Kumar Verma [email protected] · 2025
Read ADROIT: A Self-Supervised Framework for Learning Robust Representations for Active Learning by Soumya Banerjee [email protected], Vinay Kumar Verma [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07506.txt.
Introduction
Labeling data by hand is costly, so active learning picks only the most useful samples to label. Soumya Banerjee and Vinay Kumar Verma present a framework called ADROIT for that task in this 2025 paper. Good picks need a rich picture of the data, learned from both labeled and unlabeled samples. Their framework builds that picture in a self-supervised way, without extra labels.
ADROIT joins several training goals in one model based on a variational autoencoder. A helper classifier keeps the picture tied to the real task, learning from labeled data and from a self-taught signal on unlabeled data. A separate part learns to tell labeled from unlabeled samples, which guides the next picks. Knowledge from the target model is distilled back into the helper. The design aims for picks that stay sound as labeling proceeds.
Catalog record
| CREATOR | Soumya Banerjee [email protected], Vinay Kumar Verma [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07506
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ADROIT: A Self-Supervised Framework for Learning Robust Representations for Active Learning Soumya Banerjee [email protected], Vinay Kumar Verma [email protected] Abstract Active learning aims to select optimal samples for labeling, minimizing annotation costs. This paper introduces a unified representation learning framework tailored for active learning with task awareness. It integrates diverse sources, comprising reconstruction, adversarial, self-supervised, knowledge-distillation, and classification losses into a unified VAE-based ADROIT approach. The proposed approach comprises three key components - a unified representation generator (VAE), a state discriminator, and a (proxy) task-learner or classifier. ADROIT learns a latent code using both labeled and unlabeled data, incorporating task-awareness by leveraging labeled data with the proxy classifier. Unlike previous approaches, the proxy classifier additionally employs a self-supervised loss on unlabeled data and utilizes knowledge distillation to align with the target task-learner. The state discriminator distinguishes between labeled and unlabeled data, facilitating the selection of informative unlabeled samples. The dynamic interaction between VAE and the state discriminator creates a competitive environment, with the VAE attempting to deceive the discriminator, while the state discriminator learns to differentiate between labeled and unlabeled inputs. Extensive evaluations on diverse datasets and ablation analysis affirm the effectiveness of the proposed model. 1 Introduction Supervised deep learning ( Krizhevsky et al., 2012 ) has achieved considerable success, particularly in terms of image recognition ( He et al., 2016 ) ; however, the demand for extensive labeled datasets presents a formidable challenge ( Ebrahimi et al., 2017 ) . An alternative approach involves exploring various methods, such as self-supervised representation learning ( Gidaris et al., 2018 ; Chen et al., 2020 ) , active learning ( Ren et al., 2021 ) , and generative models ( Kingma & Welling, 2013 ; Goodfellow et al., 2014 ; Luo, 2022 ; Kingma et al., 2021 ) . These diverse methodologies serve distinct purposes; for example, obtaining good performance in self-supervised representation learning requires large amounts of unlabeled data. Generative models like VAEs ( Kingma & Welling, 2013 ) , GANs ( Goodfellow et al., 2014 ) , and diffusion models ( Luo, 2022 ; Kingma et al., 2021 ) are typically aimed towards generative tasks rather than representation learning. Active learning focuses on acquiring functions to minimize the number of labeled samples. This study aims to investigate the question: Given access to an expensive labeling oracle, how can we effectively reduce the need for oracle access? While the active learning paradigm addresses this by learning acquisition functions, we propose a more holistic approach. Instead of solely learning an acquisition function, we suggest considering the entire datasets, both labeled and unlabeled with access to a labeling oracle to comprehensively utilize available information, ultimately minimizing the dependency on the labeling oracle. Numerous active learning approaches, aimed at minimizing labeling costs by selecting informative samples from unlabeled pool ( Ren et al., 2021 ) , have been proposed. For instance, VAAL ( Sinha et al., 2019 ) employs both labeled and unlabeled data to model input distributions, selecting informative unlabeled samples. Yet, it is a task-agnostic approach, neglecting the conditional relation between input and output distributions. In contrast, models like TA-VAAL ( Kim et al., 2021 ) and SRAAL ( Zhang et al., 2020 ) incorporate class-conditional relationships, surpassing task-agnostic methods like VAAL ( Sinha et al., 2019 ) and LL4AL ( Yoo & Kweon, 2019 ) . Despite these advancements, current methods under-utilize available unlabeled data for task learning ( Gao et al., 2020 ; Huang et al., 2021 ; Cabannes et al., 2023 ) , missing the opportunity to enhance visual representation and model performance ( Chen et al., 2020 ) . Self-supervised learning ( Gidaris et al., 2018 ; Chen et al., 2020 ; Chen & He, 2021 ; Doersch et al., 2015 ; Noroozi & Favaro, 2016 ) exploits unlabeled data for intermediate representation learning, producing rich and semantically meaningful representations without requiring the knowledge of the annotation space (distribution). However, the integration of self-supervised learning into active learning remains under explored. Our objective is to investigate this integration not only for refining acquisition functions but also for improving the task-learner, unlocking the potential of unlabeled data for both model training and acquisition function learning. This paper introduces a novel active learning (AL) method termed A Self-Supervise D Framework for Learning RO bust Representat I ons for Ac T ive Learning (ADROIT). This approach models the input distribution by leveraging both labeled and unlabeled data, integrating self-supervision to enhance representation learning from unlabeled samples. A unified representation learner (VAE) is employed (in the same spirit as VAAL ( Sinha et al., 2019 ) ), with an added adversarial network to identify the most informative unlabeled samples. Task-awareness is achieved through a proxy task-learner, minimizing cross-entropy loss on labeled data to explicitly capture the class-conditional dependence between annotation and input labeled data. The proxy learner also minimizes self-supervised rotation loss (SSL) on unlabeled data, enriching the latent-space representation, enhancing the performance of the proxy as well as the target task-learner, ultimately benefiting both model training and acquisition function learning. Additionally, a teacher-student learning approach is implemented between the target and proxy task-learner, ensuring emulation of the target learner’s behavior. Experimental results on diverse benchmark datasets, including balanced and imbalanced ones, demonstrate the superior performance of ADROIT over recent AL baselines. Extensive ablations further confirm the significance of the proposed components. Our contributions can be summarized as follows: • The proposed model integrates SSL to include both labeled and unlabeled data in the AL model. • Utilizing a proxy task-learner or classifier, our model explicitly captures the conditional relationship between the output and input distribution. Additionally, it integrates the self-supervised rotation loss (SSL), making it an SSL-based task-aware AL approach. • We employ teacher-student learning between the target and proxy task learner. It enforces the proxy classifier to imitate the behavior of the target model while simultaneously capturing the conditional dependence and minimizing SSL. • Empirical evaluations and ablations on diverse datasets with varying class balances validate the superiority of ADROIT over existing baselines. 2 Method 2.1 Overview This section outlines the active learning (AL) setup and introduces the notation for the paper. In AL, there’s a target task (e.g., image classification) and a task-learner (T) parameterized by \zeta . Initially, a large unlabeled pool X_{U} exists, from which M samples are randomly labeled by an oracle, forming the labeled pool (X_{L},Y) . Each AL iteration selects a subset of samples with a budget (b) from X_{U} using an acquisition function and annotates them. Therefore, the labeled pool grows in size with each iteration, whereas the unlabeled pool shrinks. The task-learner (T) is trained in each AL cycle, by minimizing the sum of cross-entropy loss on labeled data and self-supervised rotation-loss ( Gidaris et al., 2018 ) on unlabeled data. Algorithm 3 illustrates the various steps of training the target task-learner (T_{\zeta}) in each stage of active learning. The proposed framework, A Self-Supervised Framework for Learning Robust Representations for Active Learning (ADROIT), is illustrated in Figure 1 . The subsequent sections provide detailed descriptions of the different components of ADROIT. 2.2 Unified Representation Learning In this work, we employ a \beta -variational autoencoder ( \beta -VAE) ( Kingma & Welling, 2013