Benchmarking and Adapting On-Device Large Language Models for Clinical Decision Support

Alif Munim, Jun Ma, Omar Ibrahim, Alhusain Abdalla, Shuolin Yin, Leo Chen, Bo Wang · 2026

Read Benchmarking and Adapting On-Device Large Language Models for Clinical Decision Support by Alif Munim, Jun Ma, Omar Ibrahim, Alhusain Abdalla, Shuolin Yin, Leo Chen, Bo Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03266.txt.

Introduction

Alif Munim and six co-authors benchmark language models that run directly on local devices for clinical decision support. Hospitals hesitate to use cloud-based systems for patient care because of privacy, yet open models small enough to run locally are often assumed to be weak. The team tests on-device models from several families across three clinical tasks, from general diagnosis to ophthalmology management and expert-style grading.

Across the tasks, the on-device models performed comparably to, and sometimes better than, much larger systems including DeepSeek-R1 and GPT-5-mini. Fine-tuning two of the local models on general diagnostic data improved them further. The study suggests privacy-preserving, offline clinical support is more capable than its reputation, an important consideration for settings with limited connectivity or strict data rules.

Catalog record

CREATORAlif Munim, Jun Ma, Omar Ibrahim, Alhusain Abdalla, Shuolin Yin, Leo Chen, Bo Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03266

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking and Adapting On-Device Large Language Models for Clinical Decision Support Alif Munim, Jun Ma, Omar Ibrahim, Alhusain Abdalla, Shuolin Yin, Leo Chen, Bo Wang Abstract Large language models (LLMs) have rapidly advanced in clinical decision-making, yet the deployment of proprietary systems is hindered by privacy concerns and reliance on cloud-based infrastructure. Open-source alternatives allow local inference but often require large model sizes that limit their use in resource-constrained clinical settings. Here, we benchmark two on-device LLMs, gpt-oss-20b and gpt-oss-120b, across three representative clinical tasks: general disease diagnosis, specialty-specific (ophthalmology) diagnosis and management, and simulation of human expert grading and evaluation. We compare their performance with state-of-the-art proprietary models (GPT-5 and o4-mini) and a leading open-source model (DeepSeek-R1), and we further evaluate the adaptability of on-device systems by fine-tuning gpt-oss-20b on general diagnostic data. Across tasks, gpt-oss models achieve performance comparable to or exceeding DeepSeek-R1 and o4-mini despite being substantially smaller. In addition, fine-tuning remarkably improves the diagnostic accuracy of gpt-oss-20b, enabling it to approach the performance of GPT-5. These findings highlight the potential of on-device LLMs to deliver accurate, adaptable, and privacy-preserving clinical decision support, offering a practical pathway for broader integration of LLMs into routine clinical practice. Introduction Large language models (LLMs) are rapidly transforming the landscape of clinical medicine. Trained on massive corpora of general-domain and biomedical text, these models have demonstrated emergent reasoning abilities that enable comprehensive summaries from medical dialogue [ 11 ] , disease diagnosis, and treatment planning [ 3 ] . For example, Med-PaLM [ 7 , 8 ] and AMIE [ 4 , 10 ] achieved near-clinician performance in open-ended medical question answering, differential diagnosis, and patient consultation simulations. In real-world evaluations [ 2 ] , LLM-assisted physicians exhibited improved diagnostic accuracy and management decisions across diverse clinical scenarios. Together, these advances illustrate the growing potential of LLMs to augment clinical expertise and support decision-making at the point of care. Despite these achievements, the translation of LLMs into real-world clinical workflows remains limited. Most frontier LLMs are proprietary, cloud-hosted, and trained on non-transparent datasets, creating challenges related to data privacy, regulatory compliance, reproducibility, and cost. The transmission of patient information to external servers conflicts with data-governance policies in many health institutions. Moreover, the high cost associated with developing, training, and deploying large-scale proprietary models can be prohibitive for many healthcare institutions. The open-source community has developed increasingly capable models that approach the performance of proprietary systems while allowing full local control. Models such as DeepSeek-R1 [ 1 ] have demonstrated competitive reasoning and clinical comprehension across diagnostic and treatment-related tasks [ 9 , 6 ] . However, the adoption of DeepSeek-R1 in clinical environments is constrained because of the large model size of 671 billion parameters, which requires substantial computational costs. The models of this scale not only hinder deployment in clinics with limited access to computing resources, but also make fine-tuning for adaptation in evolving clinical domains prohibitively expensive. These limitations underscore the need for smaller, efficient models that retain clinical reasoning ability while remaining feasible for on-premise use. The recent advancements in on-device LLMs offer a promising solution to the challenges posed by both large proprietary and open-source models. In particular, the gpt-oss family [ 5 ] represents this new generation of efficient and privacy-preserving architectures. The gpt-oss-20b model and its larger counterpart, gpt-oss-120b, are designed and optimized for deployment on a single consumer GPU with 16GB and 80GB memory, respectively. This study aims to determine whether on-device LLMs can be practically used for clinical decision support. We systematically benchmark gpt-oss-20b and gpt-oss-120b models across three representative tasks: general disease diagnosis, (ophthalmology) specialty-specific disease diagnosis and management, and simulation of human clinician judgement and evaluation (Fig. 1 ). To establish a clear performance landscape, we compare these on-device models against leading proprietary models (GPT-5 and o4-mini) and a strong open-source model (DeepSeek-R1). Fig. 1: Overview of the benchmark framework. This study compares the on-device LLMs with state-of-the-art open-source and proprietary LLMs across general disease diagnosis, specialty diagnosis and treatment recommendations on ophthalmology multiple-choice questions, and judgment for open-ended clinical decision questions. Results Dataset and evaluation methods We mainly focus on assessing the performance of LLMs on disease diagnosis, treatment recommendations, and simulating expert judgment for open-ended questions, as these are common tasks in clinical practice. We curated three datasets and benchmarked model performance on three scenarios: LLM-as-a-generalist, LLM-as-a-specialist, and LLM-as-a-clinical-judge (Figure 1 ). Specifically, to assess the capability of LLMs for general disease diagnosis, we collected 207 case reports from the Eurorad library where each case contains clinical history, findings from associated medical images (e.g., computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound), and a fine-grained differential diagnosis list (Methods). The task is to select the diagnosis from the list based on patient history and imaging findings (Supplementary Prompt 1). The second dataset was from a ophthalmology multiple-choice question dataset [ 12 ] , aiming to evaluate the specialty-specific performance of LLMs (Methods). The dataset contains 39 diagnosis questions and 91 management questions. Each question contains patient sex, age, and examinations findings, such as visual acuity, intraocular pressure, and fundus examinations, and a list of five to nine answer options. The task is to select correct answers, and each question may have multiple correct answers (Supplementary Prompt 2). The third dataset was adapted from the existing benchmarking study of the DeepSeek-R1 model [ 6 ] , including 1315 cases with human expert scores from 125 patients across five specialties (internal medicine, neurology, surgery, gynecology, and pediatrics) (Methods). LLMs are tasked to simulate human experts to assess given diagnoses from existing LLMs and assign a score from 1 to 5 based on a predefined rubric. We compare these LLM-generated scores against ground-truth human expert scores (Supplementary Prompt 3-4). For the LLM-as-a-generalist and LLM-as-a-specialist tasks, the evaluation was based on exact match accuracy For the LLM-as-a-clinical-judge task, we computed the relative error between the clinicians’ scores and the LLM’s scores. To ensure rigorous zero-shot evaluation and prevent data leakage, all three benchmark datasets were curated from cases released after the training data cutoff for all evaluated LLMs. Fig. 2: Zero-shot and fine-tuning performance of on-device LLMs. a, Results of LLM-as-a-generalist: diagnosis accuracy on a wide range of radiological cases (N=207). L, M, and H denote low, medium, and high reasoning efforts, respectively. b, Results of LLM-as-a-specialist: accuracy on ophthalmology cases (N=130) with diagnosis and management multiple-choice questions. c, Results of LLM-as-a-clinical-judge: violin plots comparing the relative error for disease diagnosis and treatment open-ended question assessment (N=1315). d, Fine-tuned gpt-oss-20b (M) model outperforms proprietary (o4-mini) and open-source LLMs (DeepSeek-R1) on the disease differential diagnosis task. e, Model performance across 10 radiological sub-specialties. On-device LLMs show competitive zero-shot performance Fig. 2 a compares the diagnostic accuracy of the evaluated models for general disease diagnosis across radiological case reports (Supplementary Table 1). While the proprietary frontier model GPT-5 achieved the highest overall average performance with an accuracy of 88.9% (95% CI: 83.9–92.5%), the on-device models demonstrated strong competitiveness relative to the established baselines. Specifically, the gpt-oss-120b (L) model achieved an average accuracy of 84.1% (95% CI: 78.5–88.4%), effectively matching the efficient proprietary model o4-mini (84.1%; 95% CI: 78.5–86.3%; p>0.05 ) and surpassing the leading

Read the complete text