Towards Scalable and Cross-Lingual Specialist Language Models for Oncology

Morteza Rohanian, Tarun Mehra, Nicola Miglino, Farhad Nooralahzadeh, Andreas Wicki · 2025

Read Towards Scalable and Cross-Lingual Specialist Language Models for Oncology by Morteza Rohanian, Tarun Mehra, Nicola Miglino, Farhad Nooralahzadeh, Andreas Wicki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08323.txt.

Introduction

Rohanian and co-authors present a 2025 framework for language models specialized in cancer care. Clinical oncology notes are messy, with gaps, mixed terms, and unclear wording, and general models often stumble on them. Their framework combines three tools: tuning on expert instructions, search over trusted sources at answer time, and a knowledge graph that links medical facts. The models stay small, so clinics with limited computing can run them.

The models handled tasks such as spotting cancer diagnoses in text, linking terms to standard codes, staging tumors, sorting reports by subtype, and predicting treatment response. A small set of German instructions from a Zurich hospital tested whether a little non-English data could carry skills across languages. Results were strong on entity and document tasks. The authors aim at tools that fit real hospital limits.

Catalog record

CREATORMorteza Rohanian, Tarun Mehra, Nicola Miglino, Farhad Nooralahzadeh, Andreas Wicki
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08323

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Scalable and Cross-Lingual Specialist Language Models for Oncology Morteza Rohanian, Tarun Mehra, Nicola Miglino, Farhad Nooralahzadeh, Andreas Wicki Abstract Clinical oncology generates vast, unstructured data that often contain inconsistencies, missing information, and ambiguities, making it difficult to extract reliable insights for data-driven decision-making. General-purpose large language models (LLMs) struggle with these challenges due to their lack of domain-specific reasoning, including specialized clinical terminology, context-dependent interpretations, and multi-modal data integration. We address these issues with an oncology-specialized, efficient, and adaptable NLP framework that combines instruction tuning, retrieval-augmented generation (RAG), and graph-based knowledge integration. Our lightweight models prove effective at oncology-specific tasks, such as named entity recognition (e.g., identifying cancer diagnoses), entity linking (e.g., linking entities to standardized ontologies), TNM staging, document classification (e.g., cancer subtype classification from pathology reports), and treatment response prediction. Our framework emphasizes adaptability and resource efficiency. We include minimal German instructions, collected at the University Hospital Zurich (USZ), to test whether small amounts of non-English language data can effectively transfer knowledge across languages. This approach mirrors our motivation for lightweight models, which balance strong performance with reduced computational costs, making them suitable for resource-limited healthcare settings. We validated our models on oncology datasets, demonstrating strong results in named entity recognition, relation extraction, and document classification. 1 Introduction Clinical oncology and related disciplines such as radiology or pathology often capture patient-related information in an unstructured or semi-structured way. At the same time, there is an increasing need to use real-world data to enable data-driven therapy decisions as a strategy that complements standardized evidence-based (study-informed) decision making. In the typical healthcare setting, oncologists must gather vast amounts of information from different data sources, including radiology images and reports, pathology reports, molecular analyses, clinical notes, and patient histories. They rely on these diverse sources to guide diagnosis, the assessment of prognosis and stage, and the decision on therapy. However, much of this data is in free text format within electronic health records (EHR) Pardoll 2012 ; Topol 2019 . Clinicians waste time and resources as they parse these notes by hand. This leads to slow, inconsistent, and error-prone decision-making, especially in resource-limited environments Bedogni 2009 . Natural language processing (NLP) offers tools to extract insights from free-text clinical records. Rule-based systems and machine learning methods with hand-engineered features have successfully identified entities such as diseases and treatments Alawad et al. 2020 . These methods fail to handle the nuanced language and variability of clinical data. Pretrained language models (LMs), such as BERT Devlin 2018 , BioBERT Lee et al. 2020 , and ClinicalBERT Alsentzer et al. 2019 , improve performance on tasks like entity recognition and literature mining by leveraging large biomedical corpora Gu et al. 2021 ; Huang et al. 2020 ; Rohanian et al. 2023 ; Rohanian et al. 2024a . Despite these advances, such models focus primarily on classification, lack flexible reasoning capabilities, and are limited in their ability to generate coherent text for summarization or prediction Ruder 2017 . Moreover, these models predominantly support English, overlooking the multilingual requirements of many healthcare systems. Large language models (LLMs), such as GPTs Brown et al. 2020 and LLaMA Touvron et al. 2023 , overcome some of these limitations by handling diverse tasks and adapting to new domains with minimal labeled data. Researchers have used them to summarize medical records, answer questions, and support clinical decisions Singhal et al. 2023 ; Saab et al. 2024 . General-purpose LLMs often fail in specialized fields like oncology. They lack domain-specific knowledge, produce inconsistent reasoning Wu et al. 2024 ; Hu et al. 2024 , and require substantial computational resources, which many healthcare institutions cannot afford. Lightweight models provide a practical alternative by delivering strong performance with significantly reduced resource requirements. Recent research has adapted LLMs for oncology-specific applications, often addressing single tasks such as named entity recognition (NER) or relation extraction Alawad et al. 2021 ; Zhou et al. 2022 ; Nishio et al. 2023 ; Fujimoto et al. 2023 . However, these approaches lack scalability and multilingual flexibility. Newer methods integrate biomedical corpora, retrieval mechanisms, and parameter-efficient fine-tuning to handle complex tasks. Some studies have curated large corpora (e.g., from the TCGA dataset) to build prognostic models or classify cancer subtypes, but these often rely on manual feature engineering or rule-based systems Alawad et al. 2021 . Other works used transformer-based models for TNM extraction, disease coding, or limited classification tasks Kefeli et al. 2024 . We propose an oncology-specialized NLP framework that combines lightweight models, bilingual adaptability, and advanced reasoning techniques. Given the Swiss healthcare system’s nature, incorporating German alongside English ensures the framework can address the linguistic diversity encountered in clinical practice at institutions like USZ. We curated minimal German instructions from clinical queries at the University Hospital Zurich (USZ) and systematically varied their number (100, 200, 400) to test whether small amounts of bilingual data can transfer domain-specific knowledge effectively across languages. Both bilingual adaptability and lightweight models align with our overarching goal of creating scalable NLP systems that can adapt to diverse healthcare environments, from large hospitals to resource-limited clinics. Our framework tries to solve key challenges in oncology NLP by integrating instruction tuning, retrieval-augmented generation, and graph-based reasoning. Each component targets specific issues in processing clinical data. Instruction tuning improves the accuracy of lightweight models for oncology-specific tasks. Models handle named entity recognition, relation extraction, TNM staging, and treatment response prediction with precision. Bilingual instructions in English and German align with real clinical use cases such as ICD-10 coding and treatment classification. Testing this variation reveals how small bilingual datasets transfer knowledge across languages and strengthen cross-lingual adaptability. RAG improves outputs by retrieving relevant clinical data from trusted sources. External datasets such as MIMIC-IV and curated German oncology reports add real-time context to the model’s responses. The retrieval process connects queries with factual information from oncology corpora. Using hierarchical methods, RAG retrieves critical details efficiently without overwhelming the input with unnecessary context. Graph-based reasoning ensures outputs are reliable and factually grounded. A knowledge graph integrates resources like UMLS, linking extracted entities to verified medical facts. Relationships between entities, such as treatments and stages, are organized as nodes and edges. Triple graph construction connects entities to authoritative references, reducing ambiguity and improving reasoning. This process strengthens the clinical reliability of model-generated outputs. Lightweight LLaMA variants (LLaMA-2-7B, LLaMA-3.1-8B, LLaMA-3.2-1B, and LLaMA-3.2-3B) combine these methods to balance efficiency and performance. The framework adapts to resource-limited clinical environments while maintaining high accuracy and flexibility across oncology-specific applications. Our contributions are as follows: 1. Oncology-Specialized Modeling: Lightweight models fine-tuned for oncology tasks like TNM staging, named entity recognition, relation extraction, document classification, and treatment prediction. Benchmarks include datasets like NCBI-Disease, i2b2-2010, and labeled subsets of TCGA. 2. Multilingual Adaptability: Minimal German instructions collected from USZ improve cross-lingual performance on ICD-10 coding and TNM staging. The bilingual framework supports diverse healthcare systems by addressing multilingual requirements. 3. Model Efficiency: Lightweight models such as LLaMA-2-7B deliver high accuracy with lower computational costs. This ensures advanced NLP tools remain accessible to institutions with limited resources. 4. Task Adaptability: The framework applies to diverse tasks, including relation extraction, document classification, and multilingual ICD-10 coding. Models adapt to new domains and tasks. The integration

Read the complete text