ALLabel: Three-stage Active Learning for LLM-based Entity Recognition using Demonstration Retrieval

Zihan Chen, Lei Shi, Weize Wu, Qiji Zhou, Yue Zhang · 2025

Read ALLabel: Three-stage Active Learning for LLM-based Entity Recognition using Demonstration Retrieval by Zihan Chen, Lei Shi, Weize Wu, Qiji Zhou, Yue Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07512.txt.

Introduction

Naming chemicals and materials inside scientific text is essential work, but labeling enough examples by hand is expensive. Zihan Chen and 4 co-authors present ALLabel in a 2025 research paper. ALLabel helps large language models do entity recognition, the task of spotting named items in text. It picks which examples are worth annotating, in three stages, using different active learning strategies in sequence.

The chosen examples become demonstrations that teach the model through in-context learning. Across three specialized domain datasets, ALLabel beat all baselines under the same annotation budget. Labeling only a small share of each dataset matched the performance of labeling everything. Ablation tests confirm each stage contributes. The result matters for labs that need accurate entity extraction but cannot afford massive annotation campaigns.

Catalog record

CREATORZihan Chen, Lei Shi, Weize Wu, Qiji Zhou, Yue Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-07512

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ALLabel: Three-stage Active Learning for LLM-based Entity Recognition using Demonstration Retrieval Zihan Chen, Lei Shi, Weize Wu, Qiji Zhou, Yue Zhang Abstract Many contemporary data-driven research efforts in the natural sciences, such as chemistry and materials science, require large-scale, high-performance entity recognition from scientific datasets. Large language models (LLMs) have increasingly been adopted to solve the entity recognition task, with the same trend being observed on all-spectrum NLP tasks. The prevailing entity recognition LLMs rely on fine-tuned technology, yet the fine-tuning process often incurs significant cost. To achieve a best performance-cost trade-off, we propose ALLabel , a three-stage framework designed to select the most informative and representative samples in preparing the demonstrations for LLM modeling. The annotated examples are used to construct a ground-truth retrieval corpus for LLM in-context learning. By sequentially employing three distinct active learning strategies, ALLabel consistently outperforms all baselines under the same annotation budget across three specialized domain datasets. Experimental results also demonstrate that selectively annotating only 5%-10% of the dataset with ALLabel can achieve performance comparable to the method annotating the entire dataset. Further analyses and ablation studies verify the effectiveness and generalizability of our proposal. 1 Introduction With the increasing prominence of AI-driven research in all science domains, many specialized scientific discovery processes demand fast and accurate textual knowledge extraction, commonly known as the named entity recognition (NER) task. As the rapid adoption of large language models (LLMs) in mainstream NLP tasks, the performance of NER has also been boosted with LLM-based solution, normally using the fine-tuning technology Gutiérrez et al. 2022 ; Zheng et al. 2023 ; Zhang et al. 2024 ; Dagdelen et al. 2024 . However, fine-tuning incurs significant cost due to the high computational resource required for updating model weights. To this end, entity recognition by fine-tuned LLMs becomes impractical under limited budget. Most recently, in-context learning (ICL) introduces a paradigm shift compared with fine-tuning and is widely applied to many downstream NLP tasks Brown et al. 2020 ; Dong et al. 2022 . In particular, ICL with demonstration retrieval selects a small number of contextually relevant demonstrations, which are integrated into the LLM prompt and guide the model to perform inference and make predictions on the test queries for specific tasks. The demonstrations serve as examples of the task, helping LLMs to learn the input and label spaces, as well as the mapping between them Wei et al. 2023 ; Pan et al. 2023 . ICL is a remarkable capability of LLMs that reduces the reliance on large annotated training datasets and avoids the need for model weight update. The latest studies have shown that the effectiveness of ICL heavily relies on the selection of demonstrations and the accuracy of ground-truth annotations Min et al. 2022 . High-quality annotations usually outperform low-quality machine-generated annotations Liu et al. 2022 . Yet, research in many fields of the natural sciences, such as chemistry and materials science, suffers from a lack of high-quality data annotations. Manual annotations in these fields demand intensive human effort and specialized domain expertise, inducing unacceptable cost Wang et al. 2021 . Therefore, selecting the most valuable subset of data for manual annotation becomes a critical challenge to improve the LLM-based entity recognition performance under the constraint of annotation budget. To address this challenge, active learning (AL), a classical method for improving annotation efficiency through selective labeling, has garnered increasing attention Ren et al. 2021 ; Diao et al. 2024 ; Xu et al. 2024 . AL aims to achieve high model performance on a withheld test set by strategically selecting a small subset of unlabeled data for annotation. However, the current application of AL focuses primarily on tasks such as text classification and knowledge reasoning, with limited exploration in NER. Furthermore, the few existing applications in NER mostly use general-domain entity recognition datasets (e.g., CONLL Sang and De Meulder 2003 ), which typically have lower extraction difficulty than those in specialized domains Kazemi et al. 2023 . Therefore, it is crucial to better integrate AL with LLM-based entity recognition tasks to address the challenges in specialized domains, including limited pre-trained knowledge and high accuracy requirements. To enhance the entity extraction performance of LLMs with reduced annotation costs, we propose ALLabel, a selective annotation framework that integrates AL into the annotation process of LLMs. By leveraging three different active acquisition strategies, ALLabel empowers LLMs to retrieve the most informative and representative examples that benefit extraction performance. Assuming a limited annotation budget, the maximum number of samples that can be manually annotated is denoted as M . ALLabel adopts a three-stage workflow to construct an optimized retrieval corpus, each stage corresponding to a specific AL sampling strategy: diversity , similarity , and uncertainty sampling. By selecting the M most valuable unlabeled samples for annotation, ALLabel ensures efficient use of the annotation budget. We conduct experiments on three datasets related to materials science and chemistry. The experimental results demonstrate the superiority of our proposed framework, which consistently outperforms baseline methods at each pool size. Further analysis of the varying few-shot setting confirms the universality of our method. The ablation study also reveals the contribution of each component. The contribution of this work can be summarized as follows: • We introduce ALLabel, a novel framework that employs LLMs as annotators for entity recognition in specialized domains, achieving superior performance compared to baseline methods with the same annotation cost. • We conduct experiments to reveal that selectively annotating only 5%-10% subset with ALLabel can achieve the same level of extraction performance by baseline methods which annotates the entire dataset. Our result demonstrates the feasibility of selective sampling in reducing annotation costs. • To the best of our knowledge, we are the first to integrate uncertainty, diversity, and similarity sampling strategies into a unified active learning framework for LLM in-context learning. Our ablation experiments show that the combination of the three strategies outperforms any pairwise combination. 2 Related Work 2.1 In-Context Learning with Demonstration Retrieval In-context learning has gained considerable attention for its flexibility and performance-cost trade-off Brown et al. 2020 ; Liu et al. 2023 . This approach relies on providing demonstrations in the input prompt to guide LLMs’ behavior. Initially, few-shot demonstrations are randomly sampled Zhang et al. 2022 ; Chung et al. 2024 , which can be suboptimal especially when there are high variations among the test queries. An alternative is to retrieve demonstrations that are tailored to the current query. Previous work has shown that demonstration retrieval can lead to substantial improvement in the task metrics compared to randomly selected demonstrations Luo et al. 2023 ; Ye et al. 2023 . Recent studies have explored methods to identify the most informative demonstrations. One prominent line of work employs retrieval-based strategies, using trainable retrievers to source relevant examples Rubin et al. 2022 . Some studies leverage uncertainty metrics to evaluate example utility, finding that examples with low perplexity often yield superior performance Gonen et al. 2023 . Levy et al. 2023 indicates that diversity and coverage of the demonstrations are crucial when the model is unfamiliar with the output symbols space. 2.2 Active Learning for NLP Active learning aims to select the most valuable samples for annotation from an unlabeled

Read the complete text