Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework

Avi-ad Avraam Buskila · 2026

Read Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework by Avi-ad Avraam Buskila (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10535.txt.

Introduction

Buskila grades small open medical chat models on steadiness, not just scores. A model that shifts answers each time it is asked makes a shaky clinic helper. His open frame runs eight quality checks plus two repeat run steadiness checks. Across three models and 50 MedQuAD items, repeat agreement peaks at one fifth. Worse, 87 to 97 percent of outputs per model come out unique, a gap one shot tests miss.

Catalog record

CREATORAvi-ad Avraam Buskila
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21859
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10535

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework Avi-ad Avraam Buskila Abstract Incorporating large language models (LLMs) in medical question answering flows demands more than high average accuracy: a model that returns substantively different answers each time it is queried is not a reliable medical tool, regardless of its mean quality score. Online health communities such as Reddit have become a primary source of medical information for millions of users, yet they remain highly susceptible to misinformation [ 18 , 19 ] ; deploying LLMs as assistants in these settings amplifies the need for output consistency alongside correctness. We present a practical, open-source evaluation framework for assessing small, locally-deployable open-weight LLMs on medical question answering, treating reproducibility as a first-class metric alongside lexical and semantic accuracy. Our pipeline computes eight quality metrics, including BERTScore, ROUGE-L, and an LLM-as-judge rubric, together with two within-model reproducibility metrics derived from repeated inference ( N = 10 N{=}10 runs per question). Evaluating three models (Llama 3.1 8B, Gemma 3 12B, MedGemma 1.5 4B) on 50 MedQuAD questions ( N = 1,500 N{=}1{,}500 total responses) reveals that despite low-temperature generation ( T = 0.2 T{=}0.2 ), self-agreement across runs reaches at most 0.20, while 87–97% of all outputs per model are unique, a safety gap that single-pass benchmarks entirely miss. We further find that the clinically fine-tuned MedGemma 1.5 4B underperforms the larger general-purpose models on both quality and reproducibility; however, because MedGemma is also the smallest model (4B vs. 8B and 12B), this comparison confounds domain fine-tuning with model scale, and a same-size comparison is needed to isolate the effect of clinical specialization. We describe the methodology in sufficient detail for practitioners to replicate or extend the evaluation for their own model-selection workflows. All code, data processing pipelines, and experiments are released to ensure full reproducibility. The full implementation is available at https://github.com/aviad-buskila/llm_medical_reproducibility. 1 Introduction Online medical communities such as Reddit have become a primary source of health information for millions of users, yet they remain highly susceptible to the rapid spread of misinformation and unverified advice [ 18 , 19 ] . While large language models (LLMs) offer a promising opportunity to support question answering in these settings, their integration into community-driven platforms requires careful evaluation of reliability and safety. In this work, we explore the potential of small, locally deployable open LLMs as assistants in medical subreddits, with the goal of improving answer quality while mitigating the risks associated with inconsistent or misleading model outputs. The availability of capable open-weight LLMs that can run entirely on local hardware has made clinical AI accessible to resource-constrained settings where patient data cannot leave institutional boundaries. Models such as Llama 3.1 [ 2 ] , Gemma 3 [ 11 ] , and domain-fine-tuned variants like MedGemma [ 7 ] can now be queried on a commodity workstation, lowering both the cost and the compliance burden of piloting LLM-based clinical decision support. Yet the dominant evaluation paradigm for these models, benchmarking single-pass accuracy against a clinically validated ground-truth corpus, is fundamentally inadequate for medical usage. A drug-interaction query that returns a correct answer 70% of the time and a plausible but wrong answer the other 30% is not a useful medical tool; neither is one that returns a different correct answer on every invocation, because the instability itself erodes trust and auditability. In safety-critical applications, regulators and risk managers need to know not only whether a model is usually correct, but how stable those answers are. Existing clinical NLP benchmarks, MedQA [ 3 ] , PubMedQA [ 4 ] , and others, evaluate single-pass accuracy and leave run-to-run variance unmeasured. This paper makes the following contributions: 1. A reproducibility-first evaluation protocol that quantifies within-model run-to-run consistency via self-agreement rate and response uniqueness rate alongside a standard accuracy suite. 2. A modular, resumable, open-source pipeline implementing the protocol with a simple three-stage CLI ( run \to score \to report ). 3. Empirical results on three small open-weight models ( N{=}1{,}500 responses across 50 MedQuAD questions), demonstrating that reproducibility failures persist even in the near-deterministic temperature regime. We also observe that the clinically fine-tuned model underperforms larger general-purpose models, though this finding is confounded by a 2–3 \times parameter count difference. 2 Related Work Self-consistency and output stability. Wang et al. [ 14 ] introduced self-consistency as a decoding strategy that samples multiple chain-of-thought paths and selects the most frequent answer, substantially improving reasoning accuracy. Chen et al. [ 15 ] extended this idea to free-form generation with universal self-consistency. Both lines of work use repeated sampling to improve output quality; our work repurposes the same repeated-sampling setup to measure output stability, turning the consistency signal itself into a first-class evaluation metric rather than a decoding technique. LLM calibration and reliability. Kadavath et al. [ 16 ] showed that language models can partially assess the correctness of their own outputs, yet their confidence estimates remain imperfectly calibrated. Our finding that low-temperature generation ( T{=}0.2 ) does not yield stable outputs complements this line of work: even when the sampling distribution is sharply peaked, the resulting text varies substantially across runs, suggesting that calibration and output determinism are distinct challenges. Medical LLM evaluation. Singhal et al. [ 10 ] demonstrated that large language models encode clinical knowledge sufficient to approach expert-level performance on medical licensing exams, while Nori et al. [ 17 ] showed that generalist foundation models can match or exceed domain-specialized systems on medical benchmarks. However, both studies evaluate single-pass accuracy and do not report run-to-run variance, leaving the reproducibility dimension of clinical LLM behavior unexplored. Our work addresses this gap by treating within-model consistency as a co-equal evaluation axis alongside accuracy. LLM-as-judge evaluation. Zheng et al. [ 13 ] established the LLM-as-judge paradigm for scalable open-ended evaluation, demonstrating strong agreement with human raters while also documenting systematic biases such as position and verbosity effects. We adopt a locally-hosted open-source judge to maintain full reproducibility and keep evaluation costs low, while acknowledging the inherent stochasticity of the judge itself as a limitation (Section 5 ). 3 Methodology 3.1 Dataset We use MedQuAD [ 1 ] , a collection of 16,412 medical question-answer pairs curated from fourteen NIH online health resources (NIHSeniorHealth, Genetics Home Reference, Cancer.gov, and others). Questions cover a wide range of clinical focus areas including symptoms, treatments, diagnoses, prevention, and prognosis. All content is in English and targeted at informed general audiences, making it a good proxy for consumer-facing clinical QA. Although MedQuAD is professionally curated rather than user-generated, its questions closely mirror the types of health inquiries commonly posted on platforms such as Reddit medical communities [ 19 ] , covering symptoms, treatments, and prevention in lay-accessible language. For reproducible sampling, we draw question subsets using a fixed random seed (seed = 42) so that any researcher can reconstruct the exact evaluation set. We evaluate N{=}50 randomly sampled questions, each subjected to 10 repeated inference runs per model, yielding 500 responses per model (1,500 total). 3.2 Models and Inference Setup We select three locally-deployable open-weight models to span a range of parameter counts and domain-specialization strategies, all served via the Ollama runtime [ 8 ] : • Llama 3.1 8B [

Read the complete text