Domain-Adapted Small Language Models for Reliable Clinical Triage

Manar Aljohani, PhD 1, Brandon Ho, MD 2, Kenneth McKinley, MD 2, Dennis Ren, MD 2, Xuan Wang, PhD 1 · 2026

Read Domain-Adapted Small Language Models for Reliable Clinical Triage by Manar Aljohani, PhD 1, Brandon Ho, MD 2, Kenneth McKinley, MD 2, Dennis Ren, MD 2, Xuan Wang, PhD 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26766.txt.

Introduction

Aljohani, Ho, McKinley, Ren, and Wang test whether open small language models can serve as sound, private decision support for ER triage. Assigning Emergency Severity Index levels from free-text triage notes remains error-prone, driving mistakes and workflow waste. Across prompting setups, they find clinical vignettes, concise summaries of triage notes, give the most exact calls, with Qwen2.5-7B balancing hit rate, steady results, and speed best. Domain tuning on expert and silver-standard pediatric triage data cuts disagreements and serious errors, beating rivals and closed large models.

Catalog record

CREATORManar Aljohani, PhD 1, Brandon Ho, MD 2, Kenneth McKinley, MD 2, Dennis Ren, MD 2, Xuan Wang, PhD 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38990
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26766

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Domain-Adapted Small Language Models for Reliable Clinical Triage Manar Aljohani, PhD 1, Brandon Ho, MD 2, Kenneth McKinley, MD 2, Dennis Ren, MD 2, Xuan Wang, PhD 1 Abstract \institutes 1 Department of Computer Science, Virginia Tech, Blacksburg, VA, US 2 Children’s National Hospital, Washington DC, USA Abstract Accurate and consistent Emergency Severity Index (ESI) assignment remains a persistent challenge in emergency departments, where highly variable free-text triage documentation contributes to mistriage and workflow inefficiencies. This study evaluates whether open-source small language models (SLMs) can serve as reliable, privacy-preserving decision-support tools for clinical triage. We systematically compared multiple SLMs across diverse prompting pipelines and found that clinical vignettes, concise summaries of triage narratives, yielded the most accurate predictions. The SLM, Qwen2.5-7B, demonstrated the strongest balance of accuracy, stability, and computational efficiency. Through large-scale domain adaptation using expert-curated and silver-standard pediatric triage data, fine-tuned Qwen2.5-7B models substantially reduced discordance and clinically significant errors, outperforming all baseline SLMs and advanced proprietary large language models (LLMs, e.g., GPT-4o). These findings highlight the feasibility of institution-specific SLMs for reliable, privacy-preserving ESI decision support and underscore the importance of targeted fine-tuning over more complex inference strategies. Introduction Triage Accuracy and Variability. Timely and accurate triage is a cornerstone of emergency department (ED) operations, influencing patient outcomes, resource allocation, and overall efficiency. The Emergency Severity Index (ESI) is a widely used five-level triage system that classifies patients by acuity and expected resource needs, ranging from ESI 1 (immediate life-saving intervention) to ESI 5 (no resources anticipated). ESI 2 includes high-risk patients without immediate resuscitation needs, while ESI 3–5 are primarily distinguished by expected resource use: ESI 3 indicates patients likely to require two or more resources, and ESI 4 those expected to require a single resource 5 . In practice, ESI assignment is often subjective and variable, especially under high workload or limited information, with studies reporting only \sim 30–50% correct triage rates 18 . For example, one multicenter study found correct assignment in just 34.1% of pediatric ED visits using ESI v4 13 , while another adult ED study reported mistriage in \sim 32% of encounters and sensitivity for high-acuity patients of only \sim 66% 14 . An international multi-center assessment reported accuracy rates near \sim 44–50% in some settings 11 . These misclassifications can delay life-saving care or overburden emergency resources, highlighting the need for reliable data-driven triage support systems. Recent work on automating ESI prediction has progressed from traditional machine learning models using structured features to deep learning methods incorporating free-text triage notes. Early approaches employed recurrent neural networks such as LSTMs to model clinical narratives 9 , followed by transformer-based encoders such as BERT and ClinicalBERT, which improved performance through large-scale pretraining and domain-specific adaptation 2 . The emergence of large language models (LLMs) marks the next stage: models like GPT-3 scale to billions of parameters and enable strong zero-shot and few-shot reasoning 3 , while medical LLMs such as Med-PaLM demonstrate robust clinical understanding without extensive task-specific supervision 15 . These advances position LLMs as promising tools for context-aware ESI prediction, particularly when annotated triage data is limited and clinical reasoning is required. Opportunities and Challenges of LLMs for Triage. Recent advances in large language models (LLMs) have demonstrated new opportunities to analyze complex unstructured clinical narratives, such as triage notes or patient vignettes 15 ; 12 ; 10 . LLMs show strong abilities in extracting clinical features, reasoning over patient narratives, and generating structured outputs while also raising important concerns about hallucination, robustness, and domain reliability 19 ; 1 . These strengths suggest potential value in assisting triage by standardizing ESI assignments, reducing cognitive load, and identifying high-risk patients earlier. However, deploying proprietary LLMs in clinical environments remains challenging. These models are typically closed-box systems, require external API calls that raise HIPAA concerns, and demand substantial compute resources that hinder on-premise deployment 17 ; 12 . Critically, their outputs are often inconsistent across repeated runs, which is problematic for safety-critical tasks such as triage that require stability, reproducibility, and transparency 6 . As a result, these limitations slow real-world operationalization despite strong clinical reasoning capabilities. Small Language Models as a Solution. Open-source Small Language Models (SLMs), typically \leq 7B parameters, offer a promising alternative for specialized, resource-constrained, and privacy-sensitive healthcare environments 16 ; 4 . Recent advances in quantization 4 , parameter-efficient fine-tuning (PEFT) 7 , and low-rank adaptation methods such as LoRA and QLoRA 4 enable efficient domain adaptation while maintaining competitive accuracy. SLMs provide several advantages: they are cost-efficient, interpretable, and adaptable to institution-specific triage data without exposing patient information 4 ; 7 ; 16 . Their open architectures support reproducibility, transparent evaluation, and community-driven improvement. Because SLMs are lightweight enough for on-premise deployment behind institutional firewalls, they support strict privacy requirements, reduce inference latency, and enable version-controlled reproducibility. Notably, SLMs often produce more stable outputs for rule-oriented clinical decision-support tasks than larger LLMs, especially when fine-tuned on structured or guideline-based workflows. In medical applications, these properties allow SLMs to achieve strong in-domain performance with significantly lower memory, compute, and operational overhead than LLMs, making them well-suited for reliable and privacy-preserving emergency triage systems. Prior work has explored similar strategies for adapting general-purpose transformers to medical question answering 15 , note summarization 8 , and radiology report generation 19 . However, few studies have examined the fine-tuning of SLMs for structured rule-based decision-support tasks such as clinical ESI triage assignment where safety, privacy, and interpretability are critical. To address this gap, we systematically evaluated multiple open-source SLM frameworks and fine-tuned a 7B-parameter SLM (Qwen2.5-7B) on de-identified retrospective emergency department triage data. We assessed the extent to which a fine-tuned Qwen2.5-7B model can approximate human ESI decision-making and analyze its error patterns (undertriage, overtriage) to identify areas for trust-aware improvement in clinical deployment. Our objective was to assess the feasibility and accuracy of small, transparent language models as a trustworthy alternative for clinical decision support in emergency triage (Figure 1 ). Figure 1: Overview of the methodological pipeline, illustrating data and training sources, silver-standard vignette generation, six prompting pipelines, QLoRA fine-tuning of Qwen2.5-7B, evaluation metrics, explainability analysis, and additional multi-agent and RAG experiments. Materials & Methods Data Sources. The dataset was obtained from Children’s National Hospital (CNH) emergency department (ED) records. An initial cohort of approximately 117,600 pediatric encounters was screened to create a structured dataset containing key triage variables for each ED visit. Each record includes a de-identified patient index, the Emergency Severity Index (ESI) assigned by clinical staff, patient age in months, and the unstructured free-text triage note describing the presenting complaint. The dataset also includes a hashed patient reference number (FIN) and the documented Chief Complaint , summarizing the primary reason for the ED visit as reported by the caregiver or patient and recorded by the triage nurse. Additional fields capture vital signs (e.g., heart rate, temperature, respiratory rate) and physical exam findings. When vital signs were unavailable, a pivot assessment field provided supplemental observations. Finally, each record contains the patient’s past medical history (PMH) as documented during triage. Inclusion/Exclusion Criteria for Our Dataset. We included encounters for individuals aged 0–21 years presenting to the pediatric emergency department who underwent standard nursing triage with an assigned

Read the complete text