KoGNER: A Novel Framework for Knowledge Graph Distillation on Biomedical Named Entity Recognition

Heming Zhang*, Wenyu Li*, Di Huang*, Yinjie Tang, Yixin Chen, Philip Payne, Fuhai Li† · 2025

Read KoGNER: A Novel Framework for Knowledge Graph Distillation on Biomedical Named Entity Recognition by Heming Zhang*, Wenyu Li*, Di Huang*, Yinjie Tang, Yixin Chen, Philip Payne, Fuhai Li† (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15737.txt.

Introduction

Heming Zhang and co-authors present a 2025 computing paper on finding named things in medical text. The task, called named entity recognition, picks out items such as diseases, drugs, and genes from writing. Standard models often fail on new topics and on rare terms in this field.

Their system, named KoGNER, draws on knowledge graphs, which store facts as linked items. First, facts from outside sources are shrunk into a light form that fits the model. Next, that knowledge is mixed into the text read by a graph network, which tracks links between items. On standard test sets, the system beat tuned models and large language models by a wide margin. The work shows that stored facts can sharpen text tools.

Catalog record

CREATORHeming Zhang*, Wenyu Li*, Di Huang*, Yinjie Tang, Yixin Chen, Philip Payne, Fuhai Li†
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15737

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

KoGNER: A Novel Framework for Knowledge Graph Distillation on Biomedical Named Entity Recognition Heming Zhang*, Wenyu Li*, Di Huang*, Yinjie Tang, Yixin Chen, Philip Payne, Fuhai Li† Abstract Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that plays a crucial role in information extraction, question answering, and knowledge-based systems. Traditional deep learning-based NER models often struggle with domain-specific generalization and suffer from data sparsity issues. In this work, we introduce K n o wledge G raph distilled for N amed E ntity R ecognition (KoGNER), a novel approach that integrates Knowledge Graph (KG) distillation into NER models to enhance entity recognition performance. Our framework leverages structured knowledge representations from KGs to enrich contextual embeddings, thereby improving entity classification and reducing ambiguity in entity detection. KoGNER employs a two-step process: (1) Knowledge Distillation, where external knowledge sources are distilled into a lightweight representation for seamless integration with NER models, and (2) Entity-Aware Augmentation, which integrates contextual embeddings that have been enriched with knowledge graph information directly into GNN, thereby improving the model’s ability to understand and represent entity relationships. Experimental results on benchmark datasets demonstrate that KoGNER achieves state-of-the-art performance, outperforming finetuned NER models and LLMs by a significant margin. These findings suggest that leveraging knowledge graphs as auxiliary information can significantly improve NER accuracy, making KoGNER a promising direction for future research in knowledge-aware NLP. Figure 1: Overall architecture of KoGNER 1 Introduction The rapid evolution of large language models (LLMs) has led to a paradigm shift in information extraction Wang et al. 2023 . With models like GPT-4o OpenAI 2024 and emerging open-source alternatives like LLaMA 3.3 Meta 2024 , DeepSeek v3 DeepSeek-AI 2024 , etc., it is now possible to extract an unlimited variety of entity types using natural language instructions - a capability that traditional NER systems, limited to a fixed set of predefined entities, cannot match Kim et al. 2024 , except for GLiNER Zaratiana et al. 2023 in a zero-shot setting. However, the promise of powerful LLM models is tempered by practical challenges: Their enormous parameter counts necessitate vast computational resources, which in turn escalate costs when deployed at scale. Conventional NER methods, including rule-based systems and machine learning models, deliver high accuracy, clear interpretability, and efficient computational performance Zhou et al. 2022 , especially in practical application scenario Xie et al. 2021 . Yet expanding their scope traditionally demands labor-intensive data annotation and retraining. Recent advances have sought to reconcile these divergent approaches. Innovations such as SCANNER Ok et al. 2024 , KCL Zhang et al. 2024b , UniNER Zhou et al. 2024 , and KAWR He et al. 2020 demonstrate that integrating structured external knowledge with the adaptive capabilities of NER systems can substantially enhance the performance and flexibility of entity and relation extraction. In addition, entity and relation extraction also provides the basis for some high-level applications such as Knowledge Graph construction Burns et al. 2024 . Knowledge distillation has emerged as a powerful technique for bridge-building the gap between high-performance, large-scale models and the practical demands of resource-constrained applications. This approach enables the student model to approximate the performance of the teacher while significantly reducing computational and memory overhead. Existing research Zhou et al. 2021 has shown that integrating CRF cross-entropy with fuzzy learning effectively facilitates the transfer of knowledge from large pre-trained models to lightweight NER systems. SKD-NER Chen and He 2023 also integrates reinforcement learning into the knowledge distillation process within a span-based continual learning framework. However, they lack domain-specific knowledge and cannot generalize to new entity types in a zero-shot fashion. Within the biomedical domain, the intricate structure of electronic patient records and clinical data, the nested organization of named entities, and the abundance of specialized terminology poses distinct challenges for NER task. The intricate biological mechanism also poses difficulties for entity relationship extraction. Recent advances in the use of LLM both directly Xiao et al. 2023 and indirectly Luu and Buehler 2024 to extract biomedical information have shown significant promise in overcoming these challenges by providing structured information curated by humans or finetuning with domain specific dataset. Here, we offer an approach that uses the best sides of both human-curated knowledge graphs and language model’s textual representation through a knowledge distillation method. This approach synergizes traditional NER techniques with the dynamic extraction capabilities of knowledge graph distillation. To the best of our knowledge, this is the first work to employ both knowledge graphs and language models to construct a comprehensive database for biomedical NER. Our framework integrates text-attributed knowledge graphs specifically tailored to address the unique challenges of biomedical text. By reimagining the role of knowledge graphs in NER, we advance the state-of-the-art in biomedical domain-specific information extraction. 2 Method This section introduces KoGNER, a model designed to enhance biomedical named entity recognition (NER) by leveraging distilled knowledge, spatial, and logical information from a large biomedical knowledge graph. By embedding these structured insights into a bi-encoder framework Zaratiana et al. 2024 , KoGNER ensures rich, context-aware entity representations, improving both accuracy and generalizability across diverse biomedical domains. 2.1 Texutal BiEncoder Given text input \mathcal{X}=[x_{1},x_{2},\cdots,x_{t},\cdots,x_{T}] and corresponding entity set \mathcal{N}=[n_{1},n_{2},\cdots,n_{c},\cdots,n_{C}] , where T is the number of tokeinized words and C is the count of entities. The span indices are produced by establishing a search window of a predetermined width w , with \mathcal{I}=\{(0,0),(0,1),\cdots,(t,t+1),\cdots,(T-1,T+w-1)\}=\{i_{pq}\}_{0}^{T\times{w}} . Afterwards, the input text \mathcal{X} and entities \mathcal{N} are embedded into their latent representations \mathcal{X^{\prime}}=\{{x^{\prime}}_{p}\}_{0}^{T}\in{\mathbb{R}^{T\times{d}}} and \mathcal{N^{\prime}}=\{n^{\prime}_{p}\}^{C}_{0}\in{\mathbb{R}^{|\mathcal{B}|\times{d}}} , where \mathcal{B} denotes the union of type categories associated with the entities. The encoded input text data, \mathcal{X^{\prime}} is then combined with span index set \mathcal{I} to yield a span representation \mathcal{S}=\{s_{pq}\}_{0}^{T\times{w}}\in{\mathbb{R}^{(T\times{w})\times{d}}} computed via s_{pq}=\text{FFN}(x^{\prime}_{p}\otimes{x^{\prime}_{q}}) with FFN indicating a two-layer feedforward neural network and \otimes representing the concatenation. Concurrently, the entity representations \mathcal{N^{\prime}} will also be encoded via the two-layer FFN. 2.2 Language Loss To evaluate whether a span (p,q) corresponds to entity type c , we calculate the following matching score: \phi(p,q,b)=\sigma(s^{T}_{pq}{n^{\prime}}_{b})\in\mathbb{R} (1) where \sigma is the sigmoid activation function. Since we use binary cross-entropy loss during training, \phi(p,q,b) represents the probability that the span (p,q) is of type b . Thus, the language loss \mathcal{L}_{\text{lang}} comprising span index set \mathcal{I} and entity types \mathcal{B} , is defined as: \displaystyle\mathcal{L}_{\text{lang}}= \displaystyle-\sum_{a\in{\mathcal{I}}\times{\mathcal{B}}}\Big(\mathbb{I}_{a\in{\mathcal{P}}}\log\phi(a) \displaystyle\quad+\mathbb{I}_{a\in\mathcal{N}}\log(1-\phi(a))\Big) (2) where the variable a represents a pair of span/entity type, such as (p,q,c) , and \mathbb{I} is an indicator function, which returns 1 when the specified condition is true and 0 otherwise for belonging to positive samples. This loss function corresponds to binary cross-entropy. 2.3 Distillation Loss During knowledge distillation, we designate the teacher model as the embedding derived from the pretrained graph neural network (GNN) and knowledge graph encoder models. Together with input tex , we add knowledge entities input as \mathcal{V}_{sub}=\{(i_{1},v^{sub}_{1}),\cdots,(i_{c},v^{sub}_{c}),\cdots,(i_{C},v^{sub}_{C})\} , where i_{c} correpsonds certain span index i_{pq} , and v_{c} denotes the corresponding entity in the knowledge graph, \mathcal{G}=\{\mathcal{V,E}\} , where |\mathcal{V}|=M and |\mathcal{E}|=e . For each entity in the knowledge graph, it has the corresponding name descriptions, denoted as z and correpsonding entity types, denoted as y . For those named description, it will be encoded by combining those name descriptions for each entity with z^{\prime}\in{\mathbb{R}^{d^{\prime}}} . And all of the

Read the complete text