Assessing the capability of large language models in answering pediatric critical care board-style questions Authors: Daniela Chanci, Ronald Moore, Henry P Foote, Matthew A Goldstein, Karan R Kumar, Alexandre T Rotta, Christoph P Hornik, Marybeth Burriss-West, Makenzie Hamilton, Noah Stanco, Monica Alvarez, Andrew Brown, Margot Johnson, Rishikesan Kamaleswaran Published: 2026 ABSTRACT AbstractThe potential of Large Language Models (LLMs) in medicine is often linked to massive, resource-intensive models. However, their practical application in specialized fields like pediatric critical care requires exploring the capability of more efficient, locally-deployable open-source alternatives. In this study, we evaluated the accuracy and clinical reasoning of open-source LLMs of varying sizes, specifically assessing if smaller, efficient models can perform comparably to larger ones on pediatric critical care multiple-choice questions. A set of 100 pediatric critical care MCQs across six clinical domains, i.e. calculation, diagnosis, ethics, management, pharmacology, and physiology, was curated by two pediatric specialists to evaluate eight open-source LLMs, ranging from 2 to 70 billion parameters. The LLMs were assessed using the overall and category-specific accuracy and clinical reasoning quality score based on a 5-points Likert scale. Additionally, six pediatric critical care fellows completed the MCQs for comparison. Cochran’s Q test, McNemar’s test, the Friedman test, Cohen’s kappa, and Fleiss’s kappa were used for the statistical analysis. While the largest model (Llama-3.3-70B) achieved the highest accuracy (78%; 95% CI, 69%-86%), a key finding was the performance of the much smaller, 14.7-billion parameter Phi-4. This efficient model was strikingly comparable, with 75% accuracy (95% CI, 65%-83%) and a similar reasoning score (4.40 vs. 4.49/5). Both models’ performance was comparable to that of the pediatric critical care fellows included in this study. The LLMs showed a strong performance in ethics but struggled with calculations. Inter-rater reliability was excellent for the clinical reasoning assessment (κ = 0.92). Our findings demonstrate that smaller, efficient LLMs can approach the performance of much larger models and pediatric critical care fellows for complex pediatric critical care reasoning. These results support further investigation for developing secure, locally-deployable decision support tools without relying on massive, proprietary systems. At the same time, these models hold potential as complementary resources for trainee education in pediatric critical care. However, their identified weaknesses, especially in calculations, pose a prohibitive barrier that underscores the need for rigorous, domain-specific validation followed by evaluation in more realistic clinical scenarios to ensure safe use in both clinical and educational contexts.Supplementary InformationThe online version contains supplementary material available at https://doi.org/10.1038/s41598-026-57353-0.Keywords: Large language models, Pediatric critical care, PICU, Multiple-choice questions, Clinical question-answering, BenchmarkSubject terms: Computational biology and bioinformatics, Health care, Mathematics and computing, Medical research Introduction Large language models (LLMs) have gained immense popularity due to their reasoning and problem-solving capabilities across a wide range of complex tasks1. In healthcare, numerous studies have evaluated the performance of LLMs for clinical documentation, notes summarization, medical question-answering, decision support based on electronic health record (EHR) data, and as complementary tools for medical education and training2–4. However, the majority of these efforts focused on adult populations, with limited assessment of LLMs capabilities in pediatrics. This gap is further amplified in pediatric critical care due to several challenges including the inherent physiological variability across age groups, differences in disease presentation, and the limited availability of high-quality datasets5. Prior work has focused on evaluating LLMs for specific pediatric tasks such as medication dosage calculation6, differential diagnosis generation7, chronic disease explanation8, Emergency Severity Index (ESI) prediction9, question-answering and education for patients and parents10,11, and general pediatric multiple-choice questions (MCQs)12. However, the performance of LLMs on pediatric critical care board-style questions, a highly specialized area that is likely not adequately represented in general-domain training data, is yet to be investigated. Furthermore, most existing studies evaluate proprietary LLMs such as GPT-413, which raise privacy concerns for real-world decision-support tools involving sensitive patient information, or very large models like PaLM 214, which pose significant computational challenges for deployment in clinical settings. In this work, we evaluate the ability of open-source LLMs to reason through pediatric critical care MCQs using multiple prompting strategies. Our contributions are as follows: (i) We have collaborated with clinicians to create a PICU MCQs dataset to evaluate LLMs. (ii) We comprehensively evaluated several state-of-the-art open-source LLMs for the selected MCQs. (iii) We conducted additional evaluation on the best-performing models to analyze their clinical reasoning capabilities on different pediatric critical care question categories and their performance compared to pediatric critical care fellows. Therefore, we not only assess LLMs viability for decision-support but also explore their potential role in medical education, where they are expected to provide trainees with accessible, interactive, privacy-preserving tools. We use the terms LLM and model interchangeably throughout this paper. Methods Our study evaluated the clinical reasoning capabilities of LLMs in MCQ answering tasks focused on pediatric critical care. The evaluation followed a two-phase design. In the first phase, 8 open-source LLMs were assessed across multiple prompting strategies. In the second-phase, the two top performing models were further evaluated for clinical reasoning quality, human comparison, and contamination analysis (Fig. 1). The study was conducted in accordance with the Helsinki Declaration, and ethical approval was obtained from the Institutional Review Board (IRB) of Duke University (Pro00115678) with a waiver of informed consent. Study flowchart. Dataset We collaborated with two pediatric critical care specialists to curate an MCQ dataset to ensure clinical relevance. Each question was carefully selected and adapted from existing question banks, including PREP ICU, Multiprofessional Critical Care Review and Assessment, and Pediatric Cardiology Board Review15–17, to reflect real-world decision-making scenarios commonly encountered in the pediatric intensive care unit (PICU). The dataset consists of MCQs that cover core topics in pediatric critical care including mechanical ventilation strategies, sepsis management, congenital heart disease, and pediatric pharmacology. Each MCQ follows a standardized format, consisting of three components: (i) a fundamental clinical knowledge question or a PICU realistic case-based scenario; (ii) four to five answer options; and (iii) a single correct response. The questions are categorized into six major domains: calculation, diagnosis, ethics, management, pharmacology, and physiology. To assess the risk of data contamination, a completion-based analysis was conducted for the top-performing models (see Supplemental Methods). Overall study pipeline. In the initial evaluation, all LLMs were assessed for zero-shot and few-shot learning, with reasoning and Chain-of-Though (CoT), with and without Retrieval Augmented Generation (RAG). The overall accuracy was used as the evaluation metric, and the two best-performing models were selected for the in-depth evaluation, which focused on zero-shot prompting strategies. It assessed the category-specific accuracy, clinical reasoning quality by two pediatric specialists, and human comparison with six pediatric critical care fellows. In addition, a contamination analysis was conducted. Multiple statistical analyses were conducted in both phases. Large language models We evaluated eight open-source, general-purpose, instruction-tuned LLMs with potential for future adaptation and deployment in clinical settings. The models were Gemma-2-2B18, Gemma-2-9B18, Gemma-3-27B19, Llama-3.1-8B20, Llama-3.2-3B20, Llama-3.3-70B20, Phi-3.5-mini21, and Phi-4 (Table 1)22. We intentionally selected LLMs spanning a wide range of model sizes and training corpus sizes to assess how scale impacts pediatric critical care reasoning performance23. Also, proprietary models were excluded from this study to focus on locally deployable models that can preserve patient data privacy and ensure model transparency. Implementation details are found in Supplemental Methods. Table 1Large language models description.ModelParameters, BContext Length – Tokens, KTraining Data Size – Tokens, TGemma-2-2B2.6182Llama-3.2-3B3.211289Phi-3.5-mini3.821283.4Llama-3.1-8B8.0312815Gemma-2-9B9.2488Phi-414.7169.8Gemma-3-27B27.412814Llama-3.3-70B70.612815Characteristics of the eight evaluated LLMs. B denotes billions, K thousands, and T trillions. Large language models description. Characteristics of the eight evaluated LLMs. B denotes billions, K thousands, and T trillions. Prompting paradigms We designed and evaluated multiple prompting paradigms for the LLMs, including zero-shot and few-shot prompting24, as well as rationale and chain-of-thought (CoT) prompting25. These paradigms were also evaluated with Retrieval Augmented Generation (RAG) to enhance prompts by incorporating relevant external context (Fig. 2)26,27. However, no fine-tuning using domain-specific data was performed in this study. Prompt templates. Each prompt typically consisted of two components: the system and the user roles. The system content included initial context, task instructions, and the specific JSON format to generate the answer. The user content included the MCQ with the corresponding answer options. Depending on the prompting paradigm, the prompt could contain additional elements, namely, the few-shot examples and the retrieved context. However, for models without native support for the system role (e.g., Gemma-2), the system content was prepended to the user content to maintain semantic structure. Zero-shot prompting In this paradigm, the models are provided with system instructions and a single MCQ. They are expected to select the correct answer based only on the question and task description, without access to any few-shot examples28. Prompt template for the rationale and chain-of-thought prompting. The main differences between the two paradigms are underlined. For few-shot prompting, the selected shots are included in the field in red. Similarly, if using retrieval augmented generation, the retrieved context is included in the field in red. Few-shot prompting This approach leverages in-context learning by providing the models with 1 to 8 few-shot examples of pediatric critical care MCQs, formatted according to the template defined in the system content24. Each example includes a question, answer options, the correct answer, and a brief explanation or step-by-step reasoning generated by GPT-4. In this regard, GPT-4 was specifically prompted to generate explanations to support clinical training given a question, choices, and correct answer. All completions were reviewed by the team of pediatric specialists to ensure correctness and clinical accuracy. For consistency across model assessment, the same set of few-shot examples was used in all test MCQs. Rationale prompting In this strategy, the LLMs are prompted to select the correct answer for a given MCQ and provide a brief explanation. Rationale prompting was applied in both zero-shot and few-shot paradigms, with or without RAG. Chain-of-thought prompting Here, the models are asked to reason through each answer option, and then select the correct one. This approach promotes reasoning, and as with rationale prompting, it was evaluated in zero-shot and few-shot settings, with and without and RAG25. Retrieval augmented generation Retrieval Augmented Generation (RAG) gained its popularity due to its ability to improve LLM performance without the need for fine-tuning26. RAG provides an LLM with knowledge relevant to the question as part of the input prompt. The process of constructing a RAG system involves three major steps, namely breaking down the collected data into chunks, vectorizing the data chunks, and storing the vectors in a vector database. For our RAG system development, we used a standard reference pediatric critical care textbook29. We employed the recursive chunking method available in the Langchain Python library package to obtain chunk sizes of 200 tokens. The 3 most relevant vectors were retrieved using the gte-base-en-v1.5 multi-lingual embedding model and cosine similarity30. More details on the RAG pipeline and implementation are included in Supplemental Methods. Finally, we conducted a qualitative analysis of zero-shot incorrect completions to identify the failure modes and facilitate a better interpretation of the RAG results. Evaluation metrics We assessed the performance and clinical reasoning capabilities of the models in pediatric critical care MCQ using two metrics. First, we calculated the accuracy, defined as the proportion of the questions for which the model selected the correct answer. This was computed using the answer choice included in the correct_answer field of the JSON-formatted model output. Accuracy was evaluated across the eight models and all prompting strategies included in this study. Second, to quantitatively evaluate the clinical reasoning quality, a pediatric intensivist and a pediatric cardiologist independently rated the reasoning included in the brief_explanation or step-by-step fields using a 5-point Likert scale based on correctness and clinical relevance as follows: 1 – Poor: The reasoning is totally incorrect, misleading, and/or irrelevant. 2 – Fair: The reasoning contains significant clinical inaccuracies. 3 – Acceptable: The reasoning is clinically accurate, but it is not related or relevant for the evaluated MCQ. 4 – Good: The reasoning is clinically accurate and relevant, but it is missing important information that should have been included. 5 – Excellent: The reasoning is clinically accurate, relevant, complete, and clear. For this assessment, we randomly selected a stratified sample of 40 questions to ensure representation across different MCQs categories. Additionally, we focused on two models representing different scales: Phi-4 (small size), and Llama-3-70B (large size). We evaluated exclusively the zero-shot prompting strategy, with rationale and CoT prompting, without RAG, to examine the models’ inherent clinical reasoning capabilities using only their pretrained knowledge, beyond simply identifying the correct answer. Human comparison To provide a preliminary clinical reference point and an exploratory human benchmark, six pediatric critical care fellows completed the MCQs used for LLM evaluation independently. These fellows were selected across different years in their training program, specifically, we included two first-year, three second-year, and one third-year. They answered the questions without time constraints or external resources, mimicking the zero-shot, rationale prompting strategy used for the LLMs. Statistical analysis The Cochran’s Q test was used to determine the performance differences across models and prompting strategies, followed by pairwise McNemar’s test with Benjamini-Hochberg adjustment when significant. Clinical reasoning scores were analyzed using the Friedman test. Agreement for the Likert ratings and the MCQs responses was quantified using Cohen’s kappa and Fleiss’s Kapa, respectively. Confidence intervals were calculated using the Wilson Score method. Results We created a dataset of 100 MCQs in pediatric critical care, each with 4 to 5 answer choices. The questions covered 6 different clinical categories: calculation, diagnosis, ethics, management, pharmacology, and physiology. Among these, calculation and ethics were the least represented, with 5 and 3 questions respectively, while management (32 questions) and physiology (31 questions) were the most prevalent categories (Supplemental Table 1). Out of the 100 questions, 8 were randomly selected to serve as few-shot examples, and the remaining 92 comprised the test set used to assess the LLMs. All models were evaluated based on the accuracy of the selected answer choice for pediatric critical care MCQs. Table 2 presents the overall accuracy of the eight LLMs assessed under zero-shot learning, comparing both rationale and CoT prompting strategies, with and without RAG. Overall, larger models performed better. For instance, Llama-3.3-70B, the largest model assessed, consistently achieved the highest performance across all prompting strategies, with accuracies ranging from 74% (68 of 92; 95% CI, 64%-82%) to 79% (73 of 92; 95% CI, 70%-86%), marginally higher than Gemma-3-27B ( p=0.077). Notably, Phi-4 despite having substantially fewer parameters (14.7B), achieved an accuracy on par with Llama-3.3-70B ( p=0.606), and slightly higher than Gemma-3-27B, although this difference did not reach statistical significance ( p=0.191). Llama-3.1-8B and Gemma-2-9B demonstrated comparable performance ( p=0.623), with Llama-3.1-8B achieving accuracy scores between 65% (60 of 92; 95% CI, 55%-74%) tand 70% (64 of 92; 95% CI 60%-78%). Meanwhile, Gemma-2-2B, the smallest model evaluated, exhibited the lowest performance, with accuracy scores below 44% across all prompting strategies. These scores were significantly lower than those of Llama-3.2-3B ( p<0.01) and Phi-3.5-mini ( p<0.01), the second and third smallest models, respectively. Additionally, the smaller-scale models, i.e. those with less than 10 billion parameters, showed an apparent drop in performance when using CoT prompting and RAG, whereas the three largest models tended to improve with CoT and, to a lesser extent, with RAG. However, these differences in performance were not statistically significant for any model ( p>0.05). In the case of Phi-3.5-mini, an apparent effect was observed, but pairwise comparison showed it was not significant ( p>0.05). Table 2Zero-shot accuracy results.ModelRationaleRationale RAGChain-of-ThoughtChain-of-Thought RAGP-valueGemma-2-2B43.4841.3043.4840.220.908Llama-3.2-3B60.8763.0452.1752.170.091Phi-3.5-mini67.3966.3056.5257.610.037Llama-3.1-8B69.5765.2266.3066.300.717Gemma-2-9B67.3964.1361.9665.220.630Phi-4 72.83 73.91 75.00 77.17 0.677Gemma-3-27B68.4869.5769.5771.740.843Llama-3.3-70B 73.91 75.00 78.26 79.35 0.303Zero-shot prompting overall accuracy for rationale and chain-of-thought prompting, with and without RAG, for the eight studied LLMs. The best performance for each setting is bolded, and the second-best is underlined. Zero-shot accuracy results. Zero-shot prompting overall accuracy for rationale and chain-of-thought prompting, with and without RAG, for the eight studied LLMs. The best performance for each setting is bolded, and the second-best is underlined. To have a better understanding of the inconsistent performance gains observed with RAG, we conducted a qualitative analysis for the zero-shot scenario. Two specific failure modes were identified for incorrect completions, retrieval error and reasoning error. In the first, the content retrieved by the RAG system, although related to the question, was only partially relevant and not useful to obtain the correct answer, biasing the model toward an incorrect option in some cases. For example, this was observed in a question about the cause of muscle soreness in an adolescent with no fractures following extubation after a motor vehicle collision. While the retrieved context included intubation-related complications, it did not contain information about neuromuscular blocking agents and their side effects, which was the information required to choose the correct answer. In the second failure more, the retrieved content was relevant to select the correct option, but the models failed to apply it appropriately. Such reasoning errors were more commonly observed on the smaller models. One example was a question requiring the calculation of cerebral perfusion pressure from provided hemodynamic values in a 12-year-old patient. The retrieved context explicitly included that the cerebral perfusion pressure is calculated as the difference between mean arterial pressure and the highest value among central venous pressure and intracranial pressure. However, the models chose the wrong answer after failing to apply the provided formula correctly. Additionally, we observed a degradation in the output format among small models when combining CoT with RAG. Figure 3 shows the overall accuracy of each evaluated model using few-shot prompting. This experiment assessed in-context learning capabilities for rationale and CoT prompting, with and without the use of RAG, across 1 to 8 few-shot examples. Gemma-2-9B exhibited improved performance when 5 or more examples were included for CoT prompting with RAG. However, overall, there is no consistent or significant gain in accuracy associated with the included few-shot examples ( p>0.05). Performance variability was significant among smaller models, such as Phi-3.5-mini and Llama-3.1-8B ( p<0.01), but no in-context learning benefits were observed. In contrast, larger models like Gemma-3-27B and Llama-3.3-70B demonstrated a more stable accuracy across few-shot settings ( p>0.05) but showed no clear trend of improvement. These results suggest that the assessed LLMs do not substantially benefit from in-context learning for a highly specialized task like pediatric critical care MCQ. Few-shot results. Based on the results across prompting strategies, Phi-4 and Llama-3.3-70B consistently outperformed the other LLMs. Therefore, subsequent analyses focused exclusively on these two models using zero-shot prompting combined with rationale and CoT prompting, without RAG. As depicted in Fig. 4a, Phi-4 and Llama-3.3-70B achieved comparable performance across all dataset categories. An apparent difference was only observed in the calculation category, where Llama-3.3-70B outperformed Phi-4. Additionally, Llama-3.3-70B demonstrated a slight improvement in accuracy for diagnosis, management, and physiology with CoT prompting compared to rationale prompting. This improvement was even more noticeable for the calculation category. However, these differences were not statistically significant ( p>0.05). Both models achieved the highest accuracy in the ethics category. Phi-4 and Llama-3.3-70B comparison. Few-shot prompting accuracy scores across prompting strategies for the eight evaluated models. Figure 4b shows the average clinical reasoning score, based on the predefined Likert scale, for Phi-4 and Llama-3.3-70B across categories. These scores were assigned by two pediatric specialists for a stratified subsample of 40 questions for the zero-shot prompting with rationale and CoT prompting, without RAG. Inter-rater agreement was excellent, with a Cohen’s Kappa of 0.92. The ethics category achieved a score of 5.0 across models and prompting strategies, consistent with the accuracy observed in Fig. 4a. Overall, accuracy and reasoning scores aligned well across categories. In the calculation category, Llama-3.3-70B achieved higher accuracy, yet both models received comparable reasoning scores with rationale prompting. In contrast, for pharmacology, Llama3.3-70B received perfect clinical reasoning scores despite slightly lower accuracy. This small difference reflects that accuracy was computed on the full test set (n = 92), while reasoning scores were limited to the subsample. CoT prompting yielded apparent better clinical reasoning, with average scores (SD) of 4.49 (1.01) and 4.40 (1.02) for Llama-3.3-70B and Phi-4, respectively, compared to 4.34 (1.16) and 4.33 (1.25) under rationale prompting. However, these differences were not statistically significant ( p>0.05). Phi-4 and Llama-3.3-70B category-specific results for zero-shot prompting with rationale and chain-of-thought prompting, without RAG. (a) Performance per dataset category. (b) Average clinical reasoning quality scores per dataset category. A qualitative analysis of the completion errors observed across categories revealed three common error patterns. First, in calculation errors, the models were able to recognize the relevant concepts but failed to apply the correct formula, and in the cases where they applied the correct formula, they failed to use the correct provided values. Second, in the management and diagnosis categories, the LLMs often applied reasonable general clinical principles that were not appropriate for the specific pediatric scenario. Third, in the management and physiology categories, when more than one option was clinically plausible, the models usually failed to select the most appropriate choice, or the most urgent required intervention. Six pediatric critical care fellows achieved an average accuracy of 68.7% and a moderate inter-subject agreement with a Fleiss’s Kappa of 0.49 (Table 3). When compared to the results obtained for Llama-3.3-70B and Phi-4 using zero-shot, rationale prompting, no significant differences were observed ( p>0.05). Therefore, the top open-source models achieved performance comparable to the pediatric critical care fellows on the assessed board-style questions. Table 3Category-specific accuracy results.SubjectYearCalculationDiagnosisEthicsManagementPharmacologyPhysiologyFellow 1275.0070.59100.0076.6788.8965.52Fellow 2350.0088.2366.6786.6777.7879.31Fellow 31100.0052.9466.6760.0066.6768.97Fellow 4175.0064.71100.0063.3333.3362.10Fellow 5250.0058.82100.0070.0066.6758.62Fellow 6275.0070.5966.6770.0066.6765.52Phi-4-25.0076.47100.0073.33100.0065.51Llama-3.3-70B-50.0076.47100.0070.0088.8972.41Category-specific accuracy comparison for six pediatric critical care fellows and the two best-performing LLMs. Category-specific accuracy results. Category-specific accuracy comparison for six pediatric critical care fellows and the two best-performing LLMs. Finally, the contamination analysis conducted for these two models showed no clear evidence of MCQ memorization from pretraining data, given the low overlap observed between the original questions and the generated completions (see Supplemental Fig. 1). For Phi-4, the mean (SD) ROUGE-L F1 was 0.073 (0.033), the mean LCS ratio was 0.022 (0.019), and the max longest continuous match was 4 words. Similarly, for Llama-3.3-70B, the mean (SD) ROUGE-L F1 was 0.103 (0.042), the mean LCS ratio was 0.034 (0.028), and the max longest continuous match was 5 words. No 8-gram overlap was found for either model. A closer inspection of the MCQ completions with higher indices of overlapping showed similarities due to common medical terminology, e.g., “medication is being administered”, “breaths per minute”, “pleural effusion”, and “the patient”, rather than memorized question content. Discussion In this study, we conducted a comprehensive evaluation of eight open-source LLMs on answering MCQs in pediatric critical care using accuracy and clinical reasoning quality scores. Our findings provide valuable insights into the potential and limitations of commonly used LLMs in this highly specialized and time-sensitive medical area that would benefit from reliable decision-support and educational tools but remains understudied. Our results for the zero-shot prompting showed a positive correlation between model size and performance. That is, in general, larger models like Gemma-3-27B and Llama-3.3-70B achieved significantly higher accuracy scores than the smaller assessed models. Previous studies have demonstrated this tendency in both general-domain tasks and medical question-answering for MedQA, MedMCQA, and PubMedQA31,32. This is reasonable as these models not only have more parameters, but their training data size is substantially larger, and therefore, they possess more foundational knowledge. However, the Phi-4 model, which is a relatively smaller model (14.7B), exhibits performance comparable to Llama-3.3-70B, and achieves marginally better accuracy than the larger Gemma-3-27B. This remarkable performance in comparison to bigger models reinforces previous findings by Abdin et al. showing Phi-4 strong reasoning capabilities, particularly in mathematical tasks22. This can be accredited to the model’s architecture and the training strategy, which leveraged synthetic and high-quality data along with a post-training alignment process22. These results suggest that besides model size, the quality and relevance of training data, and the model design and architectural choices are critical for highly specialized domains. There might be a better representation of pediatric critical care knowledge in the pre-training corpus of Phi-4, even if it is significantly smaller than the pre-training corpus of Llama-3.3-70B and Gemma-3-27B. Regarding the different prompting strategies in the zero-shot setting, our evaluation showed that CoT prompting yielded a marginal improvement in performance of larger models but had a slight negative impact on smaller models (< 10B) including Gemma-2-2B, Llama-3.2-3B, Phi-3.5-mini, Llama-3.1-8B, and Gemma-2-9B. This aligns with prior work by Wei et al. and Kojima et al., who used multiple general-domain benchmark reasoning tasks to demonstrate that CoT prompting enhances reasoning capabilities as model size increases25,28. In other words, models must be sufficiently large to effectively leverage the step-by-step reasoning process that CoT encourages. Notably, Phi-4 was the only relatively smaller model to marginally benefit from CoT prompting likely due to model’s architecture, training data, and training strategies used for its development. These findings show that smaller models may require domain-specific fine-tuning to benefit from this prompting strategy22. Similarly, RAG did not produce consistent performance improvements for smaller models and, in some cases, led to a slight performance degradation. This suggests that introducing external information may overwhelm the reasoning capabilities of smaller models, resulting in confusion rather than improved performance. In contrast, larger models demonstrated marginal gains, indicating a better ability to integrate retrieved content. Interestingly, Phi-4 again deviated from this pattern, showing marginal improvements with RAG. The conducted qualitative analysis of RAG failure modes enabled the interpretation of these inconsistencies. In this regard, identified retrieval errors may have been linked to the use of a single pediatric critical care textbook for the RAG system, that while comprehensive, might be insufficient in terms of the clinical knowledge required to successfully answer the MCQs across different categories. Additionally, the 200-token chunk size, while optimal among the configurations we tested, might be limited for complex clinical context. The reasoning errors, which were more common among smaller models, combined with the observed format degradation support that the attention capabilities of smaller models are limited. Therefore, a more tailored and sophisticated retrieval mechanism, like the medical graph-based RAG proposed by Junde Wu et al., along with more comprehensive PICU resources, might be necessary to incorporate relevant contextual information that benefits the LLMs for pediatric critical care MCQ answering33. In-context learning, evaluated via few-shot learning across different prompting strategies, was largely ineffective for this task. Even though an improvement in performance is typically expected in general domains and more general medicine tasks24,34–37, for the highly specialized and complex nature of pediatric critical care, few-shot learning might be insufficient to effectively enhance model performance. To correctly answer the evaluated MCQs, the models require specific foundational knowledge in pediatric critical care that cannot be extrapolated or inferred from more generic clinical knowledge available in the pretraining corpus. Hence, the inclusion of few-shot examples may guide the model to select an answer and provide an explanation following the desired format, but it cannot compensate for the missing PICU knowledge. Also, the evaluated questions covered a variety of topics across multiple categories, which could not be effectively represented by the 1–8 included examples. Additionally, the nuanced age-related physiologic differences found in pediatric patients might further limit the in-context learning capabilities, for example, the treatment suggested for an adolescent might not be appropriate for an infant with similar conditions. Our findings suggest that relevant domain-specific, high-quality PICU knowledge is required in the pre-training corpus of the models to improve their performance. Our results for the category-specific assessment of Phi-4 and Llama-3.3-70B with zero-shot prompting with rationale and CoT prompting, without RAG, demonstrated strong alignment between accuracy and clinical reasoning capabilities based on the Likert scale. This is particularly important for potential real-world clinical applications in pediatric critical care since the models are selecting the correct answer through correct, relevant clinical reasoning. Furthermore, both models achieved comparable performance for both evaluation metrics despite the significant difference in their sizes, with a notable exception in the calculation category, where both achieved lower results, but Llama-3.3-70B outperformed Phi-4. For this category, both models struggled applying the appropriate formula and selecting the correct values to input. Such calculation errors could be life-threatening and lead to patient harm; hence, they constitute a prohibitive barrier to unsupervised clinical deployment and use of these LLMs. Similar difficulties in mathematical reasoning within the context of medical calculations were identified by Khandekar et al.38. Conversely, both models showed a strong performance in ethics, although these results should be interpreted cautiously given the small sample size of both the ethics and calculation categories. This contrast in performance could be explained by the different capabilities and knowledge required to achieve satisfactory results in each category. More specifically, the ethics questions rely on general, high-level clinical principles that are likely well represented in the pre-training data and can be transferred from broader medical and even non-medical contexts. The calculation category requires access to the exact formula, either from the pre-training corpus or the provided context, and requires a multi-step reasoning process that includes selecting the age-appropriate formula, using the correct values, and accurately performing the calculation, making it more prone to errors. Importantly, for several questions with lower clinical reasoning scores, the models used a generally correct principle that was inappropriate for the described pediatric patient, who actually required a different management strategy due to their distinct physiology. These findings further underscore the need to incorporate domain-specific data into the pre-training corpus to improve model performance in pediatric critical care. Recent studies have demonstrated that knowledge gaps in clinical applications can be addressed with continuous pre-training and fine-tuning of general-purpose LLMs39,40. In the case of pediatric critical care, such effort would require the creation of a carefully curated, high-quality PICU-specific corpus, which is challenging considering the scarcity and lack of public availability of these specific pediatric resources. It is worth mentioning that even after domain-specific fine-tuning, some limitations may persist, e.g., calculation errors, due to broader weaknesses exhibited by these models, such as the constrains in mathematical reasoning38. Llama-3.3.-70B and Phi-4 achieved accuracies that were not significantly different to the six pediatric critical care fellows included in our human comparison, when using zero-shot learning with rationale prompting strategy. Importantly, the contamination analysis conducted for these two models showed no signs of memorization of the evaluated questions, as evidenced by near zero n-gram overlap and ROUGE-L F1 scores below 0.11, which suggests that the models relied on pediatric medical knowledge and clinical reasoning to answer the MCQ questions. These results provide a preliminary benchmark for the potential utility of open-source models for well-defined tasks and warrant further investigation as potential decision-support tools and complementary educational resources for pediatric critical care training programs. Finally, it is important to emphasize that our LLM evaluation is based exclusively on board-style MCQs, which allowed us to conduct an initial standardized assessment of the models’ foundational knowledge in a highly specialized clinical subspecialty like pediatric critical care. However, this framework constitutes a simplified evaluation that provides all the required information upfront, and the task is limited to correctly selecting from a predefined set of options and providing an accurate explanation. On the other hand, bedside PICU decision support requires a broader and more sophisticated set of skills that involves gathering relevant patient information, analyzing patient history to assess physiologic temporal progression, recommending appropriate age-specific treatment, and remaining robust to differences in clinician communication styles. Recent studies have demonstrated this disconnect and showed that performance on MCQs may lead to an overly optimistic clinical utility assessment41–43. Importantly, our findings should be interpreted as an evaluation of clinical reasoning in a standardized benchmark, rather than evidence of readiness for bedside decision support. Our work has some limitations: 1. Our 100 MCQ dataset, while carefully created by pediatric specialists, is relatively small. A larger dataset could be a better and more comprehensive representation of pediatric critical care knowledge, and thus, benefit the assessment and strengthen our findings. Additionally, our contamination analysis was conducted as an empirical approach to evaluate possible memorization, but we cannot confirm that the MCQs were excluded from the pretraining corpora since they are not publicly available. 2. Our evaluation focused exclusively on MCQs, which might not accurately reflect the complexity of real-world PICU clinical reasoning and decision-making. The models were not evaluated on realistic, open-ended, multi-step tasks, and the results should not be interpreted as direct evidence of readiness for bedside PICU decision support. 3. The clinical reasoning quality evaluation was conducted by two pediatric specialists. Similarly, the MCQ completion for human comparison was performed by six pediatric critical care fellows, which constitutes a small sample and limits the generalizability of this preliminary benchmark. Having a larger panel of clinicians across different training levels and institutions would benefit the robustness and reliability of our assessment in this context. 4. We focused on open-source models below 70B parameters, excluding widely used larger proprietary models like GPT-4, Claude 3.7 Sonnet44, and PaLM 2, which might have a different performance for this task. However, our focus on the evaluated open-source models was deliberate, given their potential for real-world clinical applications where privacy and transparency are critical concerns45,46. Despite these limitations, we demonstrated that smaller models like Phi-4 are promising for the development of pediatric critical care decision-support tools in different settings, including resource-constrained environments. In this regard, although smaller models may still require dedicated computational infrastructure, their hardware requirements make their local deployment more feasible than that of larger models. While smaller models can run on a single GPU, such as an NVIDIA L40S 48 GB, larger models require a multi-GPU infrastructure with significantly greater memory capacity. Additionally, inference latency would need to remain in the order of seconds for these models to be suitable for clinical workflows. Importantly, prior to implementation in a clinical support role, these models must have rigorous patient-specific testing and safety-focused evaluation to ensure model output is accurate and reliable across use cases. Future research should explore fine-tuning smaller models like Phi-4 using a high-quality, PICU-specific corpus. There is a need for more sophisticated and comprehensive evaluation frameworks that includes a variety of relevant real-world PICU scenarios and clinical simulations to validate the utility of these models as clinical decision-support tools and educational resources for trainees in pediatric critical care. Additionally, any future deployment in clinical settings require human oversight and institutional safeguards to ensure safe and effective use in pediatric patient care. Conclusion We evaluated eight open-source LLMs for pediatric critical care MCQ answering and found notable differences in accuracy and clinical reasoning. The larger models, Llama-3.3-70B and Gemma-3-27B, as well as the smaller Phi-4 model, showed promising potential as decision-support tools and educational resources in the PICU. However, performance on board-style MCQs is insufficient to assess readiness for bedside clinical use, and foundational knowledge gaps remain, particularly in calculation-based tasks, which represent a prohibitive barrier to any unsupervised clinical use. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 Abbreviations Chain-of-thought Electronic health record Large language model Multiple-choice question Pediatric intensive care unit Retrieval augmented generation Author contributions RK conceived and developed the study, analyzed, and interpreted the data. DC organized the data, evaluated the models, analyzed the data, drafted and edited the manuscript. RM assisted with RAG pipeline implementation and model evaluation, helped draft and edit the manuscript. HPF and MAG collected the data, assisted with analyzing and interpreting the data, and edited the manuscript. KRK, ATR, CPH, MB, MH, NS, MA, AB, and MJ assisted with analyzing the data and helped edit the manuscript. All authors edited and approved the final version of this manuscript. Funding Dr. Kamaleswaran and Ms. Chanci are supported by the National Institutes of Health under award numbers R01GM139967. Dr. Foote is supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development under award T32HD104576. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. Data availability The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. Declarations Competing interests The authors declare no competing interests. Ethics and consent to participate All methods were carried out in accordance with relevant guidelines and regulations. The study was approved by the Institutional Review Board of Duke University. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.