Predicting Failures of LLMs to Link Biomedical Ontology Terms to Identifiers: Evidence Across Models and Ontologies
Daniel B. Hier, Steven K. Platt, Tayo Obafemi-Ajayi · 2025
Read Predicting Failures of LLMs to Link Biomedical Ontology Terms to Identifiers: Evidence Across Models and Ontologies by Daniel B. Hier, Steven K. Platt, Tayo Obafemi-Ajayi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04458.txt.
Introduction
Daniel B Hier and two co-authors ask why language models fail to attach the right codes to medical terms in a 2025 research paper. Systems like the Human Phenotype Ontology and the Gene Ontology sort medical and genetic knowledge with strict codes. Models often handle medical text well yet mislink those codes. Two strong systems, GPT-4o and LLaMa 3.1 405B, were tested across both systems. Nine possible causes were tested, covering term familiarity, code usage, word shape, and system layout.
One factor dominated: how much exposure the model had to the codes themselves. Single-factor and combined checks agreed on that ranking. Creators of medical AI should ensure code-rich training material, the evidence suggests.
Catalog record
| CREATOR | Daniel B. Hier, Steven K. Platt, Tayo Obafemi-Ajayi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-04458
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Predicting Failures of LLMs to Link Biomedical Ontology Terms to Identifiers: Evidence Across Models and Ontologies Daniel B. Hier, Steven K. Platt, Tayo Obafemi-Ajayi Abstract Large language models (LLMs) often perform well on biomedical NLP tasks but may fail to link ontology terms to their correct identifiers (IDs). We investigate why these failures occur by analyzing predictions across two major ontologies—Human Phenotype Ontology (HPO) and Gene Ontology–Cellular Component (GO-CC)—and two high-performing models, GPT-4o and LLaMa 3.1 405B. We evaluate nine candidate features related to term familiarity, identifier usage, morphology, and ontology structure. Univariate and multivariate analyses show that exposure to ontology identifiers is the strongest predictor of linking success. In contrast, features like term length or ontology depth contribute little. Two unexpected findings emerged: (1) large “ontology deserts” of unused terms predict near-certain failure, and (2) the presence of leading zeroes in identifiers strongly predicts success in HPO. These results show that LLM linking errors are systematic and driven by limited exposure rather than random variability. Encouraging consistent reporting of ontology terms paired with their identifiers in biomedical literature would reduce linking errors, improve normalization performance across ontologies such as HPO and GO, enhance annotation quality, and provide more reliable inputs for downstream classification and clinical decision-support systems. I Introduction Normalization of medical concepts to an ontology is a key aspect of the natural language processing of biomedical text. Biomedical knowledge embedded in free-text clinical notes must be transformed into a computable form to support clinical decision-making, population health management, biomedical research, and precision medicine [ 1 ] . Up to 80% of the information in electronic health records (EHRs) resides in unstructured text such as discharge summaries, radiology reports, progress notes, and consultation notes [ 2 , 3 ] . Term normalization refers to the mapping of extracted clinical phrases to standardized ontology concepts and their identifiers. This is essential for enabling downstream natural language processing (NLP) applications, data analysis, and decision support [ 1 , 4 ] . Biomedical term normalization typically consists of five sub-processes: (1) identifying terms of interest in free text, (2) extracting the terms, (3) assigning each term to a relevant ontology, (4) standardizing the term to a canonical ontology entry, and (5) linking the entry to its unique identifier (Table I ) [ 5 ] . TABLE I: Biomedical Term Normalization Step Subprocess Example 1 Identify target phrase The patient was ataxic . 2 Extract target phrase ataxic 3 Categorize Use HPO 4 Standardize term Ataxia 5 Link to ID HP:0001251 While early rule-based and traditional NLP systems showed only modest success in completing this process [ 6 , 7 ] , recent advances in large language models (LLMs) have produced dramatic gains in concept extraction and normalization performance [ 8 , 9 , 10 , 11 , 12 , 13 , 14 ] . However, reported performance may overstate effectiveness for two reasons: (1) benchmark datasets are often biased toward common or easily normalized terms [ 15 , 16 , 17 ] , and (2) evaluation metrics typically reflect aggregate performance across the entire normalization pipeline, obscuring specific sources of failure such as identifier linking errors [ 13 ] . The final step— linking an ontology entry to its correct ontology identifier —is particularly vulnerable to failure. Although these failures may appear random, emerging evidence suggests they result from systematic imbalances and deficiencies in the training corpora \cite{do2024mapping}. Building on this insight, we evaluate a broad set of linguistic, structural, and usage-based features to better understand and predict failures in normalizing biomedical ontology terms. Using predictions from two state-of-the-art LLMs (LLaMa 3.1 405B and GPT-4o), we analyze 18,988 Human Phenotype Ontology (HPO) terms and 4,023 Gene Ontology (GO) cellular component terms. We engineer candidate predictors across three dimensions: (1) corpus-based frequency features (e.g., PubMed Central (PMC) identifier counts, UniProtKB annotation prevalence), (2) morphology-based features (e.g., term length, entropy, digit prefixes), and (3) ontology structure-based features (e.g., depth, leaf status). Using univariate statistics and logistic regression, we quantify each feature’s contribution to linking accuracy. Our results show that term and identifier usage patterns in the literature are more predictive of linking performance than intrinsic structural properties of the terms or the ontology. The primary contributions of this work are: 1. We introduce a comprehensive set of term-, identifier-, and ontology-level features to predict failures of large language models (LLMs) in ontology ID linking. 2. We show that features capturing identifier familiarity ( PMC Identifiers , No Annotation , and Annotation Count ) are the strongest predictors of successful term-to-identifier linking. 3. We identify problematic terms that cluster in ontology deserts —regions of sparse or unused terms—which account for a substantial fraction of normalization errors. 4. We uncover a novel formatting artifact, the leading zero effect , illustrating how LLMs exploit tokenization patterns to improve linking accuracy. 5. We demonstrate that these findings generalize across two major biomedical ontologies (HPO and GO) and two high-performing LLMs (GPT-4o and LLaMa 3.1 405B), providing actionable insights for improving ontology annotation practices in biomedical literature. While Gene Ontology annotation has gained widespread adoption, HPO annotations in the biomedical literature have lagged behind (Table III ). II Methods Data . We analyzed 18,988 terms from the Human Phenotype Ontology (HPO) and 4,023 terms from the Gene Ontology’s Cellular Component (GO-CC) hierarchy (Table III ). Ontologies were downloaded from the National Center for Biomedical Ontology (NCBO) BioPortal in CSV and OBO formats [ 18 ] . Language Models Evaluated . To evaluate the capacity of large language models (LLMs) to accurately link ontology terms with their corresponding identifiers, we employed the open-access Meta-Llama-3.1-405B-Instruct-Turbo model via the Together AI platform ( https://www.together.ai/ ) and OpenAI’s GPT-4o accessed via https://platform.openai.com/ . The query prompt for both platforms was the same: prompt = "What is the HPO ID for {’hpo_term’}? Return only the code in format HP:1234567" Feature Construction . We constructed nine features, grouped into five conceptual categories, to investigate what properties predicted success in linking a biomedical term to its ID. These are summarized in Table II . Ontology graphs (HPO and GO) were parsed from OBO files using the obonet Python package and converted to directed acyclic graphs using NetworkX, retaining only is_a relationships. Additional graph-derived features such as In_Degree , Ancestor_Count , and Subgraph_Size were explored, but only Leaf and Depth were retained for modeling. Fig. 1: Univariate predictors of successful term-to-identifier linking for HPO. Bars show the difference in mean standardized (z-score) feature values between correctly and incorrectly linked terms. Features are sorted by absolute effect size, with the largest differences shown at the top. Positive bars (e.g., Annotation Count ) indicate higher values for correctly linked terms, while negative bars (e.g., No Annotations ) indicate higher values for incorrect links. Results are shown for GPT-4o (blue) and LLaMa 3.1 405B (orange), which demonstrate broadly similar patterns. Fig. 2: Univariate predictors of successful term-to-identifier linking for GO-CC. Bars show the difference in mean standardized (z-score) feature values between correctly and incorrectly linked terms. Features are sorted by absolute effect size, with the largest differences shown at the top. Positive bars (e.g., Annotation Count ) indicate higher values for correctly linked terms, while negative bars (e.g., No Annotations