Specialty-Specific Medical Language Model for Immune-Mediated Diseases
Veysel Kocaman, Gursev Pirge, Yigit Gul, Ace Vo, Zhenya Nargizyan, David Talby · 2026
Read Specialty-Specific Medical Language Model for Immune-Mediated Diseases by Veysel Kocaman, Gursev Pirge, Yigit Gul, Ace Vo, Zhenya Nargizyan, David Talby (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28838.txt.
Introduction
Veysel Kocaman and five colleagues build a specialty medical model for immune-mediated diseases. Pulling fine clinical information from medical notes is hard, and terms for immune-mediated and infectious diseases vary across sources, curbing general language systems. They assemble and label by hand 371 case reports with two experts, setting twelve classes covering immune-mediated and infectious cases plus symptoms. A transformer model trained on clinical-domain embeddings reaches F1 0.89, beating baseline and zero-shot rivals, and expert labels plus expert vectors prove most key for disease terminology.
Catalog record
| CREATOR | Veysel Kocaman, Gursev Pirge, Yigit Gul, Ace Vo, Zhenya Nargizyan, David Talby |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30951 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28838
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Specialty-Specific Medical Language Model for Immune-Mediated Diseases Veysel Kocaman, Gursev Pirge, Yigit Gul, Ace Vo, Zhenya Nargizyan, David Talby Abstract [[email protected]] [[email protected]] [[email protected]] [[email protected]] [[email protected]] [[email protected]] keywords natural language processing ,NLP ,named entity recognition ,NER ,large language model ,LLM ,artificial intelligence ,AI ,immune mediated diseases ,infectious diseases ,clinical text mining ,biomedical information extraction † † copyrightyear: 2026 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † address: John Snow Labs Inc., Delaware, USA 1 Introduction Clinical narratives contain details about patient symptoms, diagnostic reasoning, and therapeutic choices that are often absent from structured records. Extracting these elements reliably is essential for research and for improving how clinicians understand and manage disease. Yet immune-mediated disorders and infections are commonly described in dense, idiosyncratic free text, making automated interpretation difficult. Developing methods that can identify and structure clinically relevant information from these sources remains a central challenge in healthcare NLP. Immune-mediated and infectious diseases account for a substantial global health burden and often present with overlapping manifestations. Conditions such as rheumatoid arthritis, systemic lupus erythematosus, multiple sclerosis, and persistent infections evolve over time and require careful review of longitudinal narratives to distinguish between immune-driven activity and infectious processes. Symptoms such as inflammation, fever, or organ dysfunction occur in many of these disorders and complicate diagnosis. NLP and other AI approaches provide a way to systematically organize large volumes of unstructured text, enabling tasks such as early detection, cohort construction, and assessment of disease trajectories. Although recent progress has improved clinical NLP capabilities, general-purpose systems still struggle with the specialized terminology used in immunology and infectious disease. Variability in how conditions are described, frequent ambiguity in clinical wording, and the scarcity of high-quality annotated datasets all reduce the accuracy of existing models. To address these gaps, we assembled a corpus of 371 case reports drawn from PubMed, Europe PMC, ScienceDirect, Google Scholar, and MedRxiv. Two clinical experts annotated the dataset using the Generative AI Lab 17 , defining twelve entity types that capture immune-mediated and infectious diseases as well as related symptoms and general medical concepts. Additional paraphrased examples were created to broaden linguistic diversity and improve the model’s robustness. Using this dataset, we trained several architectures and found that a Bidirectional Long Short-Term Memory–Convolutional Neural Network with character-level representations (BiLSTM-CNN-Char) yielded the strongest performance, reaching an F1 score of 0.89. The work illustrates the importance of combining domain expertise with tailored NLP methods and offers a practical framework for extracting immunology- and infection-related information from free-text documents. Such tools can support a wide range of downstream applications, including clinical decision support, disease surveillance, and translational research. 1.1 Motivation To illustrate how the pipeline operates in practice, we applied the final NER model to a clinical narrative describing a patient with Systemic Lupus Erythematosus (SLE). SLE provides a useful test case because of its heterogeneous presentation and the variety of treatments and laboratory markers commonly documented in clinical notes. The selected text was processed through the full pipeline, beginning with entity extraction. Figure 2 (Appendix) shows the model’s output at the NER stage. The system accurately identified entities corresponding to Immune_Mediated_Disease, Symptom, Treatment, Other_Disease_Disorder, and Biomarker. Key mentions—including Systemic Lupus Erythematosus, joint pain, malar rash, mycophenolate mofetil, ANA, and anti-dsDNA antibodies—were detected with correct boundaries and labels. These results demonstrate that the model handles both common and nuanced clinical terminology and can reliably convert unstructured text into structured representations. In the next step, we applied a Spark NLP Relation Extraction model to determine how the extracted entities relate to each other. The resulting SLE Knowledge Graph, generated with NetworkX 4 and shown in Figure 3 (Appendix), maps clinically relevant associations such as Immune_Mediated_Disease–Symptom, Immune_Mediated_Disease–Treatment, and Immune_Mediated_Disease–Biomarker. The graph correctly links SLE to characteristic manifestations (e.g., photosensitivity, joint pain), comorbid conditions (hypertension, osteopenia), commonly used therapies (hydroxychloroquine, corticosteroids), and immunologic markers (ANA, anti-dsDNA). Taken together, this example demonstrates how the combined NER and relation extraction pipeline can automatically transform narrative clinical descriptions into structured, interpretable knowledge representations. These outputs can support a range of downstream applications, including decision-support tools, population of disease registries, and construction of biomedical knowledge graphs for immune-mediated disorders. The case study highlights the practical readiness of the model for real-world clinical and research workflows. 1.2 Prior Work Research at the intersection of infectious and Immune-Mediated Diseases (IMDs) and clinical NLP has expanded considerably in recent years. Many studies show that tailoring NER systems to biomedical subdomains can meaningfully improve tasks such as case identification, cohort construction, and disease surveillance 26 . Disease-focused investigations illustrate how NLP tools can capture complex IMD phenotypes from free-text sources. Remaki et al. 27 designed a multi-component NLP pipeline for immune-mediated inflammatory diseases (IMIDs) using EHR data, while Kocaman et al. 20 evaluated pretrained clinical NER models across more than 138,000 clinical notes, reporting high precision values. Architectural and linguistic adaptations have also enhanced biomedical NER in languages and subdomains with limited resources 25 ; 28 ; 3 . At a broader level, transformer-based models consistently outperform rule-driven and statistical systems across extraction, classification, and phenotyping tasks 24 . However, relatively few efforts have focused specifically on the combined landscape of infectious and immune-mediated diseases. This study addresses that gap by presenting a clinician-annotated corpus and a tailored NER model designed to improve entity recognition in immunology and infectious disease contexts. 2 Methods The development of the domain-specific NER model followed a sequential workflow that combined data collection, expert annotation, model training, and iterative refinement. Clinical narratives were gathered from several open-access biomedical sources and selected to capture a range of linguistic styles and diagnostic contexts. Annotation guidelines were prepared in collaboration with clinical specialists to standardize how diseases, symptoms, and related entities should be labeled. The model was trained using the BiLSTM-CNN-Char architecture available in Spark NLP for Healthcare, which incorporates pre-trained clinical embeddings. Performance was assessed with standard evaluation metrics—Precision, Recall, and F1—along with qualitative error review. Feedback from each training round informed revisions to both the corpus and annotation scheme, allowing the system to converge toward stable and generalizable performance. 2.1 Data Sources Clinical case reports and narrative descriptions were collected from PubMed, ScienceDirect, MedRxiv, and Google Scholar. These repositories were selected to cover peer-reviewed and preprint literature relevant to immune-mediated and infectious diseases. To address the limited availability of rare disease cases and infrequently described symptoms, we supplemented the dataset with synthetic examples designed to mimic the phrasing and narrative structure typically seen in case reports. The IMD dataset was constructed iteratively in eight batches to ensure broad coverage of immune-mediated and infectious disease narratives (Table 4 (Appendix). The seed corpus comprised 52 PubMed case reports, serving as the foundation for entity definition and guideline refinement. Subsequent batches expanded the dataset with both real-world and synthetic materials, balancing clinical diversity and entity representation. Synthetic samples were generated using multiple large language model platforms. Later batches introduced long-form and focused synthetic documents emphasizing complex or rare entities, such as Geographical_Location, Fungal_Infection, and Bacterial_Infection. In total, the dataset comprised several hundred annotated texts drawn from heterogeneous sources, ensuring linguistic variability, topic diversity, and entity richness suitable for domain-specific