The Role of Active Learning in Modern Machine Learning
Thorben Werner University of HildesheimUniversitätsplatz 1 31141 Hildesheim [email protected], Prof. Lars Schmidt-ThiemeUniversity of HildesheimUniversitätsplatz 1, 31141 Hildesheim [email protected], Dr. Vijaya Krishna YalavarthiUniversity of HildesheimUniversitätsplatz 1, 31141 · 2025
Read The Role of Active Learning in Modern Machine Learning by Thorben Werner University of HildesheimUniversitätsplatz 1 31141 Hildesheim [email protected], Prof. Lars Schmidt-ThiemeUniversity of HildesheimUniversitätsplatz 1, 31141 Hildesheim [email protected], Dr. Vijaya Krishna YalavarthiUniversity of HildesheimUniversitätsplatz 1, 31141 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00586.txt.
Introduction
Thorben Werner, Lars Schmidt-Thieme, and Vijaya Krishna Yalavarthi weigh the true worth of active learning, a method that picks which data points a person should label next. The method is much studied, yet it is seldom used outside its own field of research.
The trio blame high computing cost and small gains when labeled data are scarce. In their tests, active learning beat random choice by only a few points. Methods that make more data or learn from unlabeled data did far better. Yet when those strong methods were in place first, active learning still added a gain, so they frame it as a last step, not a first fix.
Catalog record
| CREATOR | Thorben Werner University of HildesheimUniversitätsplatz 1 31141 Hildesheim [email protected], Prof. Lars Schmidt-ThiemeUniversity of HildesheimUniversitätsplatz 1, 31141 Hildesheim [email protected], Dr. Vijaya Krishna YalavarthiUniversity of HildesheimUniversitätsplatz 1, 31141 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00586
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Role of Active Learning in Modern Machine Learning Thorben Werner University of HildesheimUniversitätsplatz 1 31141 Hildesheim [email protected], Prof. Lars Schmidt-ThiemeUniversity of HildesheimUniversitätsplatz 1, 31141 Hildesheim [email protected], Dr. Vijaya Krishna YalavarthiUniversity of HildesheimUniversitätsplatz 1, 31141 Hildesheim [email protected] Abstract Even though Active Learning (AL) is widely studied, it is rarely applied in contexts outside its own scientific literature. We posit that the reason for this is AL’s high computational cost coupled with the comparatively small lifts it is typically able to generate in scenarios with few labeled points. In this work we study the impact of different methods to combat this low data scenario, namely data augmentation (DA), semi-supervised learning (SSL) and AL. We find that AL is by far the least efficient method of solving the low data problem, generating a lift of only 1-4% over random sampling, while DA and SSL methods can generate up to 60% lift in combination with random sampling. However, when AL is combined with strong DA and SSL techniques, it surprisingly is still able to provide improvements. Based on these results, we frame AL not as a method to combat missing labels, but as the final building block to squeeze the last bits of performance out of data after appropriate DA and SSL methods as been applied. 1 Introduction Training ML models in most real use cases entails working with limited amounts of labeled data. Since labels are expensive to obtain, datasets usually are split into a small labeled pool and a much larger unlabeled pool. In this paper we provide insights about the three most researched techniques to train strong models under these constraints: data augmentation (DA), semi-supervised learning (SSL) and active learning (AL). Even though all three techniques work differently (DA increases the amount of labeled data, SSL makes use of unlabeled data and AL tries to improve the selection of points that are labeled), all of them solve the same problem of limited availability of labeled data. In this sense, AL directly competes with DA and SSL as strategies for enhancing model quality in low-label regimes. Current literature has yet provided a comprehensive study of the combined application of all three methods, researching the question of which method works best in isolation, as well as whether they can be freely combined (with each consecutive method still providing a lift). In this work, we employ two well known DA methods and a collection of well-performing AL algorithms from a recent benchmark [ 10 ] . As SSL paradigm, we chose pretraining as the most used SSL paradigm in recent literature. We pay special attention to the performance of active learning methods, as techniques like DA or SSL are very rarely used in AL literature. Figure 1: Performance of Random Sampling (RS) plus different DA and SSL methods and the best AL method without DA or SSL. Numbers in each bar indicate the percentage improvement over random sampling. As a motivating example, we are comparing random sampling with various advanced training protocols containing DA and/or SSL against the best performing AL algorithm without DA or SSL on that dataset. From Fig. 1 you can observe that active learning techniques fall behind DA or SSL methods in terms of how much lift they provide over a randomly sampled labeled set and plain supervised learning without augmentations. This elicits the question, whether AL is a useful technique to combat low data scenarios at all, or if DA and SSL already exhaust the available lifts. To the best of our knowledge, no methodological paper in AL has tested their proposed algorithm in a regime with strong DA and SSL methods and only one benchmark paper [ 7 ] provides tertiary experiments about the combination of all three methods. Considering the high computational cost of AL, a systematic study on the impact of AL on a modern training pipeline including DA and SSL provides a valuable answer to the question ”Do we need AL in modern machine learning?”. To this end, we study different combinations of DA and SSL techniques for three different datasets and test, whether AL methods can provide an additional lift on the optimal setup per dataset. 1 1 1 Code will be available under: TODO Key Insights 1. Active Learning is the least efficient method of overcoming low data scenarios 2. Despite that, active learning can still provide lifts even when paired with strong data augmentation and semi-supervised learning techniques 3. In this regime, every competitive active learning method performs exactly on par 2 Problem Description We are experimenting on pool-based AL with classification models. Mathematically we have the following: Given a dataset \mathcal{D}_{\text{train}}:=(x_{i},y_{i})\quad i\in\{1,\ldots,N\} with x\in\mathcal{X},y\in\mathcal{Y} (following [ 10 ] we similarly have \mathcal{D}_{\text{val}} and \mathcal{D}_{\text{test}} ) we randomly sample an initial labeled pool L^{(0)}\sim\mathcal{D}_{\text{train}} that we call the seed set. We suppress the labels from the remaining samples to form the initial unlabeled pool U^{(0)}=\mathcal{D}_{\text{train}}/L^{(0)} . We define an acquisition function to be a function that selects a batch of samples of size \tau from the unlabeled pool a(U^{(i)}):=\{x_{b}^{(i)}\}\in U^{(i)}\quad b:=[0,\ldots,\tau] . We then recover the corresponding labels y^{(i)}_{b} for these samples and add them to the labeled pool L^{(i+1)}:=L^{(i)}\cup\{(x_{b}^{(i)},y^{(i)}_{b})\} and U^{(i+1)}:=U^{(i)}/\{x_{b}^{(i)}\}\quad b:=[0,\ldots,\tau] . The acquisition function is applied until a budget B is exhausted. We measure the performance of a model \hat{y}:\mathcal{X}\to\mathcal{Y} on the held out test set \mathcal{D}_{\text{test}} after each acquisition round by fitting the model \hat{y}^{(i)} on L^{(i)} and measuring the test accuracy. We allow the fitting process to additionally depend on a DA technique and an SSL technique who aid model training: \text{TRAIN}(\hat{y}^{(i)},\\ L^{(i)},\text{DA},\text{SSL}) . DA is allowed to alter the labeled samples of that iteration: \text{DA}(L^{(i)}) , while SSL can make use of either a fully labeled external dataset \mathcal{D}_{\text{ext}}\cap\mathcal{D}_{\text{train}}=\emptyset and/or the unlabeled pool of our current dataset: \text{SSL}(\hat{y}^{(i)},\{\mathcal{D}_{\text{ext}},U^{(i)}\}) . 3 Related Work We are currently not aware of any methodological paper that combines strong DA and SSL with their proposed method, as they usually focus on improving upon other acquisition functions in a comparable setting. However, some recent benchmark papers have studied aspects of DA and SSL: [ 2 ] found that DA does not only improve the overall test accuracy of BADGE, but also its label efficiency. [ 10 ] test AL methods in an SSL setting by pre-encoding their datasets with a pretrained encoder, but do not employ DA in any of their experiments. [ 7 ] propose to tune DA as part of the hyperparameters in the first iteration of AL, as well as evaluating AL in two SSL scenarios. We extend the study of [ 7 ] in three ways: First, by allowing a comprehensive evaluation of DA and SSL techniques on the tested datasets in order to find the optimal combination, second, by quantifying how much each method contributes to overcoming the low data problem, and third, by significantly extending the list of tested AL algorithms. 4 Methodology This work serves as a guide for machine learning practitioners tasked with training high-performing models on unlabeled datasets. In many cases, random sampling and simple techniques like data augmentation (DA) and leveraging pretrained ImageNet weights are the only methods applied, due to their low computational cost and accessibility. We study the impact