BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Conference: P
Andrey Sakhovskiy, Elena Tutubalina · 2025
Read BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Conference: P by Andrey Sakhovskiy, Elena Tutubalina (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07588.txt.
Introduction
Andrey Sakhovskiy's 2025 paper, joined by 1 co-author, aligns language models with a biomedical knowledge graph. Biomedical language models read medical text well but grasp complex concept structures and graph-encoded facts poorly. Knowledge graphs store facts as linked concepts, and the Unified Medical Language System is the field's central example. The proposed method, BALI, trains the two kinds of model together.
During pre-training, concept mentions in text are linked to the medical knowledge graph, and the local subgraph around each concept serves as a matching partner for the mention. A dedicated graph encoder learns alongside the language model, and their representations are aligned. Applied to leading models such as PubMedBERT and BioLinkBERT, BALI improved language understanding scores and entity representation quality. Notably, gains came from minimal pre-training on a small alignment dataset drawn from PubMed abstracts.
Catalog record
| CREATOR | Andrey Sakhovskiy, Elena Tutubalina |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07588
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3729901 ISBN: 979-8-4007-1592-1/2025/07 CCS: Computing methodologies Information extraction CCS: Applied computing Health care information systems CCS: Information systems Language models Andrey Sakhovskiy, Elena Tutubalina Abstract . In recent years, there has been substantial progress in using pretrained Language Models (LMs) on a range of tasks aimed at improving the understanding of biomedical texts. Nonetheless, existing biomedical LLMs show limited comprehension of complex, domain-specific concept structures and the factual information encoded in biomedical Knowledge Graphs (KGs). In this work, we propose BALI ( B iomedical Knowledge Graph and Language Model Ali gnment), a novel joint LM and KG pre-training method that augments an LM with external knowledge by the simultaneous learning of a dedicated KG encoder and aligning the representations of both the LM and the graph. For a given textual sequence, we link biomedical concept mentions to the Unified Medical Language System (UMLS) KG and utilize local KG subgraphs as cross-modal positive samples for these mentions. Our empirical findings indicate that implementing our method on several leading biomedical LMs, such as PubMedBERT and BioLinkBERT, improves their performance on a range of language understanding tasks and the quality of entity representations, even with minimal pre-training on a small alignment dataset sourced from PubMed scientific abstracts. 1. Introduction In recent years, advancements in biomedical Natural Language Processing (NLP) have been largely driven by the development of domain-specific pre-trained Language Models (LMs) ( Alsentzer et al. 2019 ; Beltagy et al. 2019 ; Michalopoulos et al. 2021 ; Yasunaga et al. 2022b ; Gu et al. 2022 ; Mannion et al. 2023 ; Sakhovskiy et al. 2024 ) . Despite the recent success of Large Language Models (LLMs) in the general domain, they fall short of lightweight domain-specific biomedical LMs ( Gu et al. 2022 ; Yasunaga et al. 2022b ) by a large margin ( Chen et al. 2023 ; Bai et al. 2024 ) . While domain-specific models have shown remarkable performance on biomedical NLP benchmarks, for instance, on the Biomedical Language Understanding and Reasoning Benchmark (BLURB) ( Gu et al. 2022 ) , they have been shown to impose limited domain-specific factual knowledge understanding ( Sung et al. 2021 ; Meng et al. 2022 ) and information extraction ( Chen et al. 2023 ) . The concept structure and factual knowledge within a specific domain are often represented through extensive knowledge graphs (KGs), which can describe millions of domain-specific concepts and their inter-relations. A notable example in the biomedical domain is the Unified Medical Language System (UMLS) 1 1 1 https://www.nlm.nih.gov/research/umls/index.html KG ( Bodenreider 2004 ) , a comprehensive meta-thesaurus covering over 4M concept from 166 lexicons/thesauri. Recent lines of research have iteratively improved the current state-of-the-art performance on biomedical entity representations by pre-training either on UMLS concept names ( Liu et al. 2021a ; Liu et al. 2021b ; Yuan et al. 2022 ) or aligned text-KG subgraph pairs ( Sakhovskiy et al. 2024 ) . However, these work mostly fine-tuned LMs for entity linking, limiting their applicability beyond this specific task. This narrow focus can hinder the models’ ability to generalize across diverse biomedical texts and concepts. Recent efforts to improve the knowledge capabilities of LMs involve integrating text and knowledge graphs (KGs) in a shallow or one-way manner ( Zhang et al. 2019a ; Wang et al. 2021a ; Sun et al. 2021 ; Baek et al. 2023 ) (e.g., from KG to text for retrieval-augmented methods like RAG ( Lewis et al. 2020 ) , REALM ( Guu et al. 2020 ) , and REPLUG ( Shi et al. 2024 ) ), which could hinder multi-hop reasoning. Another approach is using an interaction token ( Zhang et al. 2022 ; Yasunaga et al. 2022a ) or a projector ( Tian et al. 2024 ) that depends on implicit exchanges between modalities. Unlike previous efforts, we explore the alignment of the uni-modal embedding spaces using anchors to better capture interconnected information and dependencies between textual and graph modalities. This alignment may contribute to enhanced multi-hop reasoning capabilities, as the model can more effectively traverse and reason across the aligned spaces. In this paper, we introduce Knowledge Gra ph and B iomedical Language Model A li gnment (BALI), a novel pre-training approach that enhances LM with external knowledge by concurrently training a distinct KG encoder and aligning the representations of both the LM and the graph. Specifically, as in Figure 1 , given a (text, local KG) pair, a graph neural network (GNN) is utilized to capture and encode the graph knowledge into node embeddings, while pre-trained LM is used to obtain textual entity representations. Textual entity representations and concept node representations are used as anchors to align the two uni-modal embedding spaces. In this work, we seek to answer the following research questions (RQs): RQ1: Is the proposed cross-modal LM-KG alignment procedure with explicit alignment between two representation spaces beneficial for biomedical NLP downstream tasks? RQ2: What is the most effective graph representation for LM-KG alignment? RQ3: Is the utilization of an external graph encoder more effective for cross-modal LM-KG alignment or using graph linearization followed by LM encoding is sufficient? To comprehensively assess our model, we perform extensive experiments across several benchmarks for question answering and entity linking tasks. Initially, we pretrain several LMs with BALI, leveraging the PubMed corpus and UMLS KG. Our experiments demonstrate that BALI outperforms several biomedical language models, including BioLinkBERT ( Yasunaga et al. 2022b ) and PubMedBERT ( Gu et al. 2022 ) . Specifically, PubMedBERT shows mean accuracy improvements of 2.1%, 1.7%, and 6.2% on the PubMedQA, MedQA, and BioASQ benchmarks, respectively. BALI significantly enhances the ability of LMs to generate distinguishable and informative representations of biomedical concepts. In particular, BioLinkBERT with BALI pretraining performs on par or slightly better than the task-specific SapBERT model, which is pre-trained on 12M UMLS triples (4M concept nodes). Our research highlights that our cross-modal knowledge graph alignment, applied to both text and the knowledge graph, notably enhances language-knowledge representations after a small pre-trainning stage involving 1.5M sentences and 600K nodes only. The source code as well as pre-trained models are publicly available at: https://github.com/Andoree/BALI-BERT . 2. Related Work Knowledge-Augmented Language Models One line of research on knowledge-enhanced LMs ( Liu et al. 2020 ; Sun et al. 2020 ; Ke et al. 2021 ; Mannion et al. 2023 ; Yuan et al. 2022 ; Moiseev et al. 2022 ) attempted to infuse factual information into LM input either by augmenting natural language texts with relational triples or directly training on relational triples. Various methods ( Sevgili et al. 2019 ; Zhang et al. 2019b ; He et al. 2020 ; Wang et al. 2021b ; Peters et al. 2019 ; Schildwächter et al.