Measuring Stability Beyond Accuracy in Small Open-Source Medical Large Language Models for Pediatric Endocrinology
Vanessa D’Amario, Randy Daniel, Alessandro Zanetti, Dhruv Edamadaka, Nitya Alaparthy, Joshua Tarkoff · 2026
Read Measuring Stability Beyond Accuracy in Small Open-Source Medical Large Language Models for Pediatric Endocrinology by Vanessa D’Amario, Randy Daniel, Alessandro Zanetti, Dhruv Edamadaka, Nitya Alaparthy, Joshua Tarkoff (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11567.txt.
Introduction
Vanessa D'Amario and five coauthors grade small medical models beyond accuracy. Their 2026 preprint studies pediatric endocrinology as the test field. Small open-source medical LLMs promise low-resource deployment. They also widen access where big models cannot run.
Evaluation of such models often stops at accuracy alone. Multiple choice question scores are the usual yardstick. Consistency, robustness, and reasoning behavior go unmeasured. A model can score well yet answer unsteadily in clinic use. The team measures stability as a first-class trait. Pediatric endocrinology gives a focused, expert test domain. Safer small models need such broader checks before use.
Catalog record
| CREATOR | Vanessa D’Amario, Randy Daniel, Alessandro Zanetti, Dhruv Edamadaka, Nitya Alaparthy, Joshua Tarkoff |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11567
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measuring Stability Beyond Accuracy in Small Open-Source Medical Large Language Models for Pediatric Endocrinology Vanessa D’Amario, Randy Daniel, Alessandro Zanetti, Dhruv Edamadaka, Nitya Alaparthy, Joshua Tarkoff Abstract Small open-source medical large language models (LLMs) offer promising opportunities for low-resource deployment and broader accessibility. However, their evaluation is often limited to accuracy on medical multiple choice question (MCQ) benchmarks, and lacks evaluation of consistency, robustness, or reasoning behavior. We use MCQ coupled to human evaluation and clinical review to assess six small open-source medical LLMs (HuatuoGPT-o1 ( 5 ) , Diabetica-7B ( 26 ) , Diabetica-o1 ( 26 ) , Meditron3-8B ( 20 ) , MedFound-7B ( 13 ) , and ClinicaGPT-base-zh ( 23 ) ) in pediatric endocrinology. In deterministic settings, we examine the effect of prompt variation on models’ output and self-assessment bias. In stochastic settings, we evaluate output variability and investigate the relationship between consistency and correctness. HuatuoGPT-o1-8B achieved the highest performance. The results show that high consistency across the model response is not an indicator of correctness, although HuatuoGPT-o1-8B showed the highest consistency rate. When tasked with selecting correct reasoning, both HuatuoGPT-o1-8B and Diabetica-o1 exhibit self-assessment bias and dependency on the order of the candidate explanations. Expert review of incorrect reasoning rationales identified a mix of clinically acceptable responses and clinical oversight. We further show that system-level perturbations, such as differences in CUDA builds, can yield statistically significant shifts in model output despite stable accuracy. This work demonstrates that small, semantically negligible prompt perturbations lead to divergent outputs, raising concerns about reproducibility of LLM-based evaluations and highlights the output variability under different stochastic regimes, emphasizing the need of a broader diagnostic framework to understand potential pitfalls in real-world clinical decision support scenarios. Introduction Large Language Models (LLMs) hold significant potential in healthcare, particularly to improve chronic disease management and clinical decision support ( 22 ; 14 ) . High-performance medical LLMs have showed strong results on different aspects of clinical care, from diagnostics to patient interfacing ( 21 ; 13 ; 15 ) . However, these models are often large, exceeding 70 billions parameters, and proprietary, making them inaccessible for independent validation or adaptation. They become impracticable solutions in resource-constrained or low-income healthcare settings. In contrast, small open-source medical LLMs offer the potential for lightweight, locally deployable, and transparent tools that could reduce barriers to access. Nonetheless, their reliability and reproducibility have not been systematically addressed. A critical question to address is understanding if these models, as released and without fine-tuning, encode sufficient clinical knowledge to support real-world use. Pediatric endocrinology presents a unique testbed for the evaluation of medical LLMs. Pediatric endocrinology is facing growing patient volume, longer waiting times, and a larger range of conditions in the last years ( 10 ; 3 ) . At the same time, access to pediatric endocrinology expertise is limited in many regions ( 3 ) , creating a potential role for AI-driven decision support, provided that they can deliver trustworthy outputs. In this study, we explore this question by evaluating a selection of small (under 10 billion parameters) previously fine-tuned open-source medical LLMs in pediatric endocrinology. Given the lack of evaluation of medical LLMs in this field, our work fills this gap by systematically assessing model behavior on a curated set of pediatric endocrinology cases, offering insights into both the capabilities and limitations of LLM tools in this clinical subspecialty. To evaluate model performance in pediatric endocrinology, we leverage Multiple Choice Questions (MCQs) drawn from real-world clinical content. MCQs are a widely used strategy for evaluating medical LLMs, particularly small, domain-specific models. While convenient and well-structured for automated scoring and scalable benchmarking, MCQ-based evaluation has well-documented limitations. Question difficulty varies widely, distractors may lack clinical plausibility, and accuracy alone often obscures reasoning quality, robustness, or consistency. Recent studies outside of the healthcare context ( 12 ; 9 ) have also highlighted response instability among LLMs in MCQ settings and self-assessment bias ( 28 ; 25 ; 16 ) . Despite these challenges, MCQs remain one of the most accessible formats for systematic model evaluation. In this work, we embrace their practicality while addressing their shortcomings: we propose a multidimensional evaluation framework that goes beyond standard accuracy-based evaluation and measures how stable model outputs are under clinically irrelevant input variations, parameter changes, and computational variability. Specifically, we evaluate six small medical LLMs and assess consistency , by comparing decisions across prompt variations, including syntactical variations and the removal of answer option labels. We also evaluate response consistency and correctness across stochastic sampling, where the degree of stochasticity of the inference process is determined by different temperature values and self-assessment bias , by presenting models with a choice between their own rationale and a gold-standard clinical explanation. We lastly investigate numerical stability , examining how changes in software environments affect model outputs despite identical inputs. This multidimensional analysis provides a structured diagnostic of small medical LLM readiness and exposes failure modes that accuracy metrics alone would not be able to capture. Our goal is not only to compare models but to define reproducibility-oriented evaluation criteria for trustworthy, accessible LLMs in medicine. In doing so, we highlight which models show the greatest promise for practical clinical integration and identify areas requiring further improvement. Materials and Methods Multiple Choice Questions from Pediatric ESAP 2021-2022 To evaluate the reasoning capabilities of small open-source medical LLMs we rely on the Pediatric Endocrine Self-Assessment Program 2021-2022 (ESAP) ( 18 ) . The program is a question bank containing 100 among clinical cases and factual questions based on clinical guidelines. Clinical cases items present a patient scenario, including relevant medical history, examination findings, and laboratory results, and typically ask to determine the most likely diagnosis or the best course of action in patient management. Other more factual items focus on guideline-based knowledge such as indications for specific medications, appropriate monitoring strategies, or recommended referral to a subspecialist. Each ESAP item includes a description of the clinical case, a question, five possible answers (each comes with a letter from A to E), an educational objective, the correct answer, and the clinical explanation for each answer. Drawing on the expertise of leading pediatric endocrinologists, the program is highly technical and aimed at medical specialists. The program’s learning objectives include recognizing clinical features of endocrine, growth, and metabolic disorders; applying current diagnostic and treatment options; identifying risk factors; and evaluating endocrine manifestations of systemic diseases. As such, we considered it as an optimal starting point for knowledge assessment in pediatric endocrinology. We excluded nine ESAP items from the analysis, since those rely on images and graphs. For each clinical case described, we extracted the following fields: <description> , <question> , <options> , <correct answer> , and <explanation> . Small Open-Source Medical LLMs We evaluate six small-sized, open-source medical LLMs designed for biomedical reasoning. HuatuoGPT-o1 ( 5 ) is based on the LLaMA 3.1 8B architecture ( 6 ) and is trained using continued pre-training and supervised fine-tuning in a bilingual corpus of 1.1TB (Chinese and English), with benchmarks including MedQA, MedMCQA, CMB, CMExam, and CMMLU. Diabetica-7B ( 26 ) uses the Qwen2-7B-Instruct model ( 29 ) , fine-tuned through low-rank adaptation and self-distillation on public medical QA sets, SFT