Few-Shot Large Language Models for Actionable Triage Categorization of Online Patient Inquiries
Liqi Zhou, Jiafu Li · 2026
Read Few-Shot Large Language Models for Actionable Triage Categorization of Online Patient Inquiries by Liqi Zhou, Jiafu Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15680.txt.
Introduction
Liqi Zhou and Jiafu Li test prompted large language models on triage of inquiries. The task sorts inquiries into self-care, schedule-visit, urgent review, or emergency referral. They build gold, silver, and few-shot sets of 300, 700, and 40 cases. Six models at zero, four, and twelve shots face TF-IDF and BioBERT baselines. The best model, Claude Haiku 4.5 at twelve shots, hits macro-F1 0.475, topping BioBERT at 0.378, though ranges overlap. They conclude models can aid triage and human review, but not solo deployment.
Catalog record
| CREATOR | Liqi Zhou, Jiafu Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79704 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15680
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Few-Shot Large Language Models for Actionable Triage Categorization of Online Patient Inquiries Liqi Zhou, Jiafu Li Abstract Online patient inquiries are often informal, incomplete, and written before professional assessment, yet they must still be routed to an appropriate level of clinical follow-up. We study this as a four-class actionable triage task – self-care, schedule-visit, urgent-clinician-review, or emergency-referral, and ask whether prompted large language models (LLMs) can support such routing under low-resource labeling conditions. Using the public HealthCareMagic-100K corpus, we construct a 300-example human calibrated gold evaluation set, a 700-example auto-labeled silver training set, and a 40-example few-shot pool. We compare Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT) baselines train on silver labels against six prompted LLMs under 0-shot, 4-shot, and 12-shot conditions respectively. Accordingly, we evaluate with macro- F 1 F_{1} alongside safety-aware metrics, including emergency-recall, under-triage rate, and severe under-triage rate. The strongest LLM (Claude Haiku 4.5, 12-shot) reaches macro- F 1 F_{1} 0.475, exceeding the best supervised baseline (BioBERT, 0.378) on point estimate, with overlapping confidence intervals. Few-shot prompting and two-model agreement help in label-dependent ways: self-care agreement is reliable, urgent-clinician-review is not. We conclude that LLMs can support triage prioritization and selective human review, but not autonomous deployment. 1 Introduction Online patient inquiries contain unstructured symptom descriptions, medication questions, test result concerns, and care-seeking requests. These messages are often written before professional assessment and may be incomplete, informal, or clinically ambiguous. Health systems and online care platforms, therefore, need scalable methods to help route patient-authored messages to an appropriate level of follow-up. This study focuses on actionable triage categorization rather than diagnosis or treatment recommendations. Given a patient inquiry, the task is to assign exactly one of four care-routing labels: self-care, schedule-visit, urgent-clinician-review, or emergency-referral. This framing emphasizes the operational question of what type of response is appropriate, not the clinical question of what condition the patient has. Automatic triage is challenging for three reasons. First, patient-authored text often lacks key clinical details such as duration, severity, comorbidities, vital signs, or current care status. Second, high-acuity cases are relatively rare but safety critical. Third, standard aggregate metrics such as accuracy or even macro- F_{1} may obscure clinically asymmetric errors. In triage, under-triage of urgent or emergency cases is typically more concerning than conservative over-triage, so evaluation should explicitly measure high-acuity recall, false negatives, and severity-direction errors. Large language models (LLMs) may be useful for this setting because they can interpret free-text symptom descriptions, follow label definitions, and adapt through prompting without task-specific fine-tuning. However, their value for online patient inquiry triage cannot be assumed. Prior work has studied clinical text classification, patient-message urgency, emergency department acuity assessment, and LLM-based medical natural language processing (NLP), but less is known about how few-shot LLMs behave in a low-resource, action-oriented triage setting with human-calibrated labels and safety-aware evaluation. In particular, it remains unclear whether prompted LLMs offer practical advantages over lightweight supervised baselines, how sensitive they are to prompt design, and whether model agreement can support selective human review. We address these questions using public online medical dialogue data and a guideline-informed, human calibrated gold evaluation set. We compare supervised Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT) baselines against open-source and API-served LLMs under zero-shot and few-shot prompting conditions. We evaluate models using both standard classification metrics and safety-aware triage metrics; furthermore, we analyze model disagreement as a signal for human-in-the-loop (HITL) selective review. We summarize our contributions as follows: • We formulate online patient inquiry triage as a four-class actionable routing task rather than as diagnosis generation or general medical text classification. • We construct a guideline-informed, human-calibrated gold evaluation set from public medical dialogue data, with separate silver training and few-shot prompt-construction splits. • We compare supervised TF-IDF and BioBERT baselines against multiple prompted LLMs under 0-shot, 4-shot, and 12-shot settings. • We evaluate triage performance using safety-aware metrics, including urgent and emergency recall, under-triage, severe under-triage, and high-acuity false negatives. • We explore two-model consensus as an oracle HITL selective prediction strategy, showing that model agreement is label-dependent and should not be treated as a universal auto-accept signal. 2 Related Work 2.1 Clinical Text Classification and Triage NLP Clinical text classification has long supported triage and routing tasks. In emergency departments, machine learning and clinical NLP have been applied to predict acuity from triage documentation and clinical notes 5 , and broader reviews show sustained interest while emphasizing heterogeneity in clinical settings, data sources, and evaluation designs 11 . A separate line of work moves from clinician-authored documentation to patient-authored messages: 3 study perceived severity in patient-generated telemedical COVID queries, and 12 evaluate BERT-based methods for patient-message triage in small-data settings. These studies are directly relevant because patient-authored text is informal, incomplete, and written before professional assessment, but they remain primarily supervised task-specific systems and do not address safety-aware, action oriented routing over public online medical dialogue. 2.2 LLMs in Healthcare NLP LLMs have made zero-shot and few-shot prompting a practical alternative to task-specific supervised training in healthcare NLP. However, prior evidence does not support assuming that prompted LLMs are uniformly superior. Large-scale biomedical NLP benchmarking shows substantial variation by task and evaluation design 1 , and instruction-tuned LLMs in clinical and biomedical NLP show similarly task dependent few-shot gains 6 . Comparative work on health-related text classification further supports evaluating traditional machine learning, supervised pretrained language models, and LLM prompting within a shared experimental framework 4 . This motivates our comparison among TF-IDF baselines, BioBERT, and prompted LLMs on the same human-calibrated gold set, rather than treating LLM performance as self-evidently superior. 2.3 LLMs for Triage and Urgency Evaluation LLM-based triage has recently emerged as an active research direction, particularly in emergency acuity assessment and patient-message urgency. 13 evaluate LLMs for clinical acuity assessment in emergency department settings, 9 compare LLM triage performance in emergency medicine, and large-scale retrospective ED evaluations further emphasize caution before autonomous deployment 10 . In the patient-message setting, 2 frames urgency as a pairwise ranking problem over asynchronous portal messages. Together, these studies establish that LLMs may be useful for triage-related tasks, but they also highlight that triage errors are clinically asymmetric, and more importantly under-triage of urgent or emergency cases can carry greater risk than conservative over-triage. Therefore, model evaluation should include safety-aware outcomes such as high-acuity recall, false-negative counts, under-triage, and severe under-triage, not only accuracy or macro- F_{1} . 2.4 Positioning of This Paper Prior work has treated triage primarily as emergency acuity prediction 13 ; 9 ; 10 or as pairwise urgency ranking of patient messages 2 . We instead frame it as a four-class actionable routing task that asks what response a patient inquiry warrants, not what condition it indicates. We compare supervised TF-IDF and BioBERT baselines against prompted open-source and API LLMs on a single human-calibrated gold set, and we add three further analyses: prompt sensitivity, safety-aware metrics, and two-model consensus as a selective prediction signal. The resulting system is positioned as decision support for triage prioritization and selective human review, not as a replacement for clinician judgment. 3 Data and Annotation 3.1 Source Dataset We use