Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG

2026

Read Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16422.txt.

Description

Klila and colleagues compare two ways of giving language models structured biomedical knowledge from the UMLS Metathesaurus. One route bakes a large knowledge graph into model weights through continual pretraining, producing BERTUMLS and BioBERTUMLS; the other consults the graph at answer time through GraphRAG. On biomedical understanding tasks, the baked-in version helped BERT most on knowledge-heavy questions, while gains for BioBERT were uneven. Graph retrieval, by contrast, lifted a LLaMA model on PubMedQA by over 3 points and on BioASQ by 5, with no retraining.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33507
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16422

Browse the full library · About the Tower of Babel