Efficient Human-in-the-Loop Active Learning: A Novel Framework for Data Labeling in AI Systems

Yiran Huang, Jian-Feng Yang · 2025

Read Efficient Human-in-the-Loop Active Learning: A Novel Framework for Data Labeling in AI Systems by Yiran Huang, Jian-Feng Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00277.txt.

Introduction

Yiran Huang, Jian-Feng Yang, and Haoda Fu propose, in a 2024 paper, a smarter way to choose which data experts should label for AI training. Labels are costly, above all when only trained eyes can give them, as with medical scans. Active learning picks the most useful items to label, but classic forms only ask which item comes next.

Their framework also decides how the question is asked, since query types differ in cost and yield. A model blends the yield of mixed query schemes into one plan. A data-driven balance of exploring and exploiting steers the picks. Tests on five real datasets, one a hard image task, beat rival methods on accuracy and loss. Expert hours may stretch further with such planning.

Catalog record

CREATORYiran Huang, Jian-Feng Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-00277

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Human-in-the-Loop Active Learning: A Novel Framework for Data Labeling in AI Systems Thanks: Corresponding author Yiran Huang, Jian-Feng Yang Abstract Modern AI systems rely heavily on labeled data, yet labeling is often expensive and labor-intensive, especially when requiring special skills such as reading radiology images by physicians. To most efficiently use experts’ time for data labeling, one promising approach is human-in-the-loop active learning. However, traditional active learning methods are limited to single-label queries and fail to leverage more flexible query types in many real-world settings. In this work, we propose a novel active learning framework with significant potential for application in modern AI systems. This framework incorporates different query schemes and jointly determines which question to ask and which data points to query. We also introduce a method to integrate full and partial information obtained from these diverse queries. In addition, we develop an innovative model-agnostic exploration and exploitation framework to filter out redundant samples. Experiments on five datasets, including two real-world image datasets, demonstrate that the proposed framework substantially outperforms all other methods. These results highlight the framework’s promise for applications across a range of scientific domains. Keywords: Data labeling; Dynamic prediction; Exploration and exploitation; Flexible query design; Full and partial information; Information gain 1 Introduction A diverse range of AI systems has emerged for practical applications. Large language models, such as Generative Pre-trained Transformers (GPT) ( Vaswani et al., 2017 ; Radford et al., 2018 ) , have demonstrated exceptional capabilities in natural language processing tasks, including understanding, generating, and interpreting human language. Similarly, reinforcement learning models have been successfully applied to real-world scenarios, such as autonomous driving ( Kiran et al., 2021 ) . These models are data-hungry, requiring a significant amount of labeled data for effective training ( Zhao et al., 2023 ) . This poses challenges in data collection, large-scale annotation, and data quality improvement. Across various scientific domains such as language processing, reinforcement learning, medical imaging and speech recognition, there exists a substantial volume of unlabeled data and a relatively small amount of annotated data ( Budd et al., 2021 ; Zhu and Goldberg, 2022 ) . It is prohibitive for traditional supervised model training, especially for deep learning models, because labeling a significant portion of data can be impractical. One solution to this dilemma is human-in-the-loop active learning algorithms. Active learning is a process in which humans interact with the learning system to improve its efficiency. It aims to identify the most informative unlabeled data and entrust them to oracle experts for labeling. The high quality of these selectively labeled data points significantly improves model performance. As a result, the overall labeling requirement can be reduced to a manageable level. Criteria from different perspectives are used to recognize informative data, including uncertainty-based methods ( Settles and Craven, 2008 ; Nguyen et al., 2022 ) , Bayesian approaches ( Houlsby et al., 2011 ; Kirsch et al., 2019 ) , representative-based methods ( Ash et al., 2020 ) , diversity-based methods ( Zhdanov, 2019 ) , density-based methods ( Xu et al., 2007 ) , cluster-based methods ( Wang et al., 2017 ) , disagreement-based methods ( Cohn et al., 1994 ; Hanneke, 2011 ; Houlsby et al., 2011 ; Hanneke and Yang, 2019 ) , distance-based methods ( Xing et al., 2002 ; Yang et al., 2012 ; Deng et al., 2023 ) , assembling methods ( Hsu and Lin, 2015 ) , and data-driven approaches ( Konyushkova et al., 2017 ) . In this paper, we introduce a novel human-in-the-loop active learning framework that allows not only standard label queries but also more flexible and efficient queries. We focus on uncertainty-based methods, as they are among the most widely adopted active learning strategies and can be applied across a wide range of models. Beyond selecting the most informative samples, balancing the trade-off between exploration and exploitation is crucial, as this balance helps mitigate the “cold start” problem in active learning ( Yuan et al., 2020 ; Yi et al., 2022 ) . The exploration and exploitation allocate a proportion of the budget to gather information from areas with limited knowledge, because the model is likely to be inaccurate during the early stages of active learning and may misidentify the informativeness of each data point (see, for example, Ren et al. (2021) ). To address the need for balancing exploration and exploitation in active learning, we propose a novel adaptive framework. This mechanism improves query efficiency and helps the algorithm better utilize limited annotation budgets. In this work, in addition to identifying the most informative data, we investigate different ways to annotate the data. Traditional active learning methods primarily focus on which data to query but often neglect how to label the data. Motivated by this limitation, we aim to develop a method that addresses both the challenges of data labeling and quality enhancement by considering various perspectives on labeling process. Specifically, traditional active learning algorithms query only one type of question: “what is the class of this point?” (referred to as a “Class” query) and receive a fully informative answer about the point. The differences among algorithms lie in their information measurements. In many practical scenarios, it is possible to query innovative questions such as “are all of these data from a specific class?” or “is any of these data from a specific class?” (referred to as “All” and “Any” queries, respectively). For example, in epidemiological testing, doctors conduct pathological tests to determine whether a patient has a disease and to identify the specific illness. By pooling samples from several patients and asking “does any of these patients suffer from a specific illness?”, experimenters can obtain the result with a single laboratory test ( Hogan et al., 2020 ; Song et al., 2022 ) . Extending active learning to incorporate multiple question types involves several challenges. First, the new questions may yield partial information. It is crucial to effectively leverage this information to improve model fitting. Second, with an enriched set of available questions, we must not only select specific realizations but also decide which question to query. Third, as previously noted, balancing exploration and exploitation remains a significant challenge. To address these challenges, we propose a new active learning framework whose novelty lies in the design for efficient human-in-the-loop learning. The paper first introduces a method to construct a probabilistic model using information gathered from various questions, including both full and partial information. Additionally, the paper proposes an uncertainty-based active learning framework using information gain. Unlike traditional methods that rely solely on full label queries, our framework introduces a flexible query system that supports a diverse set of questions, including full label queries and more efficient binary questions. These new questions not only reduce annotation cost, but also enhance efficiency by allowing information from multiple data points to be gathered simultaneously. The framework also automatically selects both the question and the data to be queried. In parallel, a data-driven exploration and exploitation framework is proposed. The framework can be embedded in a large number of active learning algorithms. The core idea of the framework is to automatically filter out data points likely to contain redundant information, based on their model-guided

Read the complete text