Better Generalizing to Unseen Concepts: An Evaluation Framework and An LLM-Based Auto-Labeled Pipeline for Biomedical Concept Recognition
Shanshan Liu, Noriki Nishida, Fei Cheng, Narumi Tokunaga, Rumana Ferdous Munne, Yuki Yamagata, Kouji Kozaki, Takehito Utsuro, Yuji Matsumoto · 2026
Read Better Generalizing to Unseen Concepts: An Evaluation Framework and An LLM-Based Auto-Labeled Pipeline for Biomedical Concept Recognition by Shanshan Liu, Noriki Nishida, Fei Cheng, Narumi Tokunaga, Rumana Ferdous Munne, Yuki Yamagata, Kouji Kozaki, Takehito Utsuro, Yuji Matsumoto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16711.txt.
Introduction
Shanshan Liu and a team of eight co-authors tackle a hard task in drug and health data. Tools must spot named health terms in text. New terms show up all the time, with no hand-made lists. Their 2026 preprint builds a test frame for this gap. It also builds a way to tag terms with no human hand.
Their test frame ranks terms in a tree of groups. Fresh scores gauge how well tools catch new terms. A big language model then tags text on its own. Those tags can train tools with less cost. Sound term tags feed search, care, and drug watch tasks.
Catalog record
| CREATOR | Shanshan Liu, Noriki Nishida, Fei Cheng, Narumi Tokunaga, Rumana Ferdous Munne, Yuki Yamagata, Kouji Kozaki, Takehito Utsuro, Yuji Matsumoto |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16711
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Better Generalizing to Unseen Concepts: An Evaluation Framework and An LLM-Based Auto-Labeled Pipeline for Biomedical Concept Recognition Shanshan Liu, Noriki Nishida, Fei Cheng, Narumi Tokunaga, Rumana Ferdous Munne, Yuki Yamagata, Kouji Kozaki, Takehito Utsuro, Yuji Matsumoto Abstract Generalization to unseen concepts is a central challenge due to the scarcity of human annotations in Mention-agnostic Biomedical Concept Recognition (MA-BCR). This work makes two key contributions to systematically address this issue. First, we propose an evaluation framework built on hierarchical concept indices and novel metrics to measure generalization. Second, we explore LLM-based Auto-Labeled Data (ALD) as a scalable resource, creating a task-specific pipeline for its generation. Our research unequivocally shows that while LLM-generated ALD cannot fully substitute for manual annotations, it is a valuable resource for improving generalization, successfully providing models with the broader coverage and structural knowledge needed to approach recognizing unseen concepts. Code and datasets are available at https://github.com/bio-ie-tool/hi-ald . 1 Introduction Biomedical Concept Recognition (BCR) —identifying ontology concepts expressed in free-text passages—is foundational for knowledge-intensive applications. Accurate and efficient CR systems may facilitate the construction and maintenance of structured biomedical knowledge bases, accelerate knowledge discovery, and ultimately support downstream applications such as therapeutic innovation. Most prior works formulate the BCR task as mention-based recognition: detect a text span (a “mention”) in the text and then link it to an ontology concept 9 ; 12 ; 15 ; 1 ; 6 . While being effective when concepts are explicitly expressed, this setting is less aligned with biomedical discourse, where many concepts are expressed implicitly. This misalignment persists even when mentions are annotated. For instance, in an abstract included by the HPO GSC+ corpus 11 , the span “melanin pigment synthesis in the hair, skin, and eyes” is annotated as Generalized hypopigmentation (HP:0007513) . The concept is implicit with respect to that span—it is neither the ontology label nor a semantically equivalent paraphrase—and the annotation is supported by intra-abstract cues (e.g., hypopigmentation, reduced retinal pigment) rather than surface matching. In addition, mention-based supervision requires costly two-level annotation (mention spans and mention-concept mappings), limiting scalability in domains where expert labeling is expensive. We therefore study MA-BCR , which directly identifies ontology concepts from a passage without requiring intermediate mention spans. This formulation better matches biomedical discourse and alleviates annotation burden (no need for mention spans for model training and inference). Independent of the task formulation, a key requirement for real-world deployment of recognizers is the ability to generalize to unseen concepts , given that manually labeled datasets (MLDs) cover only a small fraction of concepts in biomedical ontologies due to the high requirements on expertise and annotation time. For example, the HPO GSC+ corpus comprises 228 abstracts and covers approximately 2.4% of Human Phenotype Abnormality (HPA) concepts in the Human Phenotype Ontology (HPO) 5 . Despite its importance, this capability has been under-evaluated. Beyond results showing that recognizing unseen concepts was nearly infeasible for a recognizer MA-COIR 10 , clear experimental evidence on unseen-concept recognition remains scarce . To systematically probe model generalization in recognizing unseen concepts—where models cannot predict the target directly—we propose an innovative evaluation framework. This framework is built upon two core components. First, we develop three types of hierarchical indices (ontology-aware, semantic-based, and hybrid), while each of them can serve as a structured concept space to make generalization measurable. Second, we introduce two new metrics that operate within this space: Unseen Recall-oriented Closeness (U-RC), which quantifies how close a model gets to the unseen concept, and Unseen Candidate set Size (U-CS), which measures how much the model shrinks the search space compared to the full label sets. Our evaluation framework provides the tools to measure generalization, yet the question of how to improve it remains. A practical pathway is to expand concept coverage and contextual diversity through Auto-Labeled Data (ALD). Given their success as automatic annotation providers in other NLP tasks 13 , Large Language Models (LLMs) present a scalable avenue for higher concept coverage. This potential is especially important for MA-BCR, a task that requires good language understanding capabilities for assigning passage-level labels without relying on surface mentions. We therefore design a novel auto-labeling pipeline based on LLMs. Its components are tailored for MA-BCR, and the final design is determined through extensive empirical validation. We employ our evaluation framework to assess this data-centric approach, specifically asking: • RQ1: Is LLM-generated ALD sufficient for training a recognizer that performs effectively on both seen and unseen concepts? • RQ2: Despite the noise, does ALD improve generalization to unseen concepts? Our investigation proceeds as follows. We first quantify the quality of our ALD by benchmarking it against Manual-Labeled Data (MLD). We then employ the MA-COIR recognizer, which outputs pre-defined hierarchical concept indices as predictions, allowing for a grounded assessment of generalization. Finally, by training MA-COIR on different scales of MLD and ALD, we compare their effectiveness in enhancing model performance. On two pairs of concepts and ontologies, HPA concepts in HPO and Homeostasis Imbalance Process (HoIP) concepts in HoIP Ontology 16 , models trained on ALD consistently achieve substantial gains in unseen-oriented metrics as the data volume increases, despite exhibiting lower exact-matching accuracy than their MLD-trained counterparts. This indicates that exposure to broad and noisy supervision enhances hierarchy-aware generalization even when exact-match accuracy lags. We position LLM-based auto-labeling not as a replacement for manually labeling, but as a complementary source of coverage and structural signal for unseen concept recognition. Our contributions are as follows: • We introduce an LLM-based auto-labeling pipeline for MA-BCR, and construct two large auto-labeled datasets. To our knowledge, LLM-based automatic labeling has not been explored in MA-BCR before our work. • We design three types of hierarchical indices and two metrics for directly assessing hierarchy-aware learning and search-space reduction on unseen concepts. • Results show that models trained on large-scale ALD underperform MLD-trained models on F1 but generalize better to unseen concepts, clarifying ALD’s role in MA-BCR. Figure 1: Overview of the LLM-based Auto-labeling Pipeline. Given an input passage, the pipeline begins by generating intermediate claims (Arrow 1), followed by generating candidate concept names from the claims (Arrow 2). The resulting names are matched to ontology terms by comparing the representations (encoded by SapBERT) of generated and ontological names, forming a preliminary list of concept candidates, shown in the blue-shaded table. Then concept classification, relabeling, and guideline-based filtering steps are applied to get the final annotations (highlighted in bold). The instance will be taken as a training instance if its quality meets the requirement. 2 Related Works Biomedical Concept Recognition. Biomedical concept recognition has traditionally been approached as a two-stage task: mention detection followed by entity linking. Early systems leverage neural architectures for named entity recognition and linking 9 ; 12 , while more recent methods introduce LLM-based pipeline 15 ; 1 ; 6 ; 4 or reformat that task as a one-step sequence-to-sequence generation framework 10 . However, these approaches often struggle with concept ambiguity, limited coverage, and implicit expressions—issues exacerbated by the sparsity of high-quality annotated data. Hierarchical Indexing. Hierarchical indexing has proven effective in tasks involving large output spaces, such as extreme multi-label classification 17 ; 8 , document retrieval 14 , and biomedical concept recognition (BCR) 10 . By organizing labels or