Performance of Large Language Models in Supporting Medical Diagnosis and Treatment
Diogo Sousa, Guilherme Barbosa, Catarina Rocha, Dulce Oliveira · 2025
Read Performance of Large Language Models in Supporting Medical Diagnosis and Treatment by Diogo Sousa, Guilherme Barbosa, Catarina Rocha, Dulce Oliveira (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10405.txt.
Introduction
Diogo Sousa and co-authors grade chatbots on a national medical exam in this 2025 paper. The team sat a range of large language models, open and closed, for Portugal's 2024 specialty access test, a serious benchmark of medical knowledge. They weighed raw accuracy against running cost.
Scores varied widely, and several models beat the human student benchmark on this exam. Combining accuracy with cost singled out clear leaders. Step-by-step reasoning styles received a close look for their effect on answers. The authors position such models as promising sidekicks for clinicians facing hard decisions, provided accuracy and cost are tracked together rather than by headline scores alone.
Catalog record
| CREATOR | Diogo Sousa, Guilherme Barbosa, Catarina Rocha, Dulce Oliveira |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10405
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Performance of Large Language Models in Supporting Medical Diagnosis and Treatment Diogo Sousa, Guilherme Barbosa, Catarina Rocha, Dulce Oliveira Abstract The integration of Large Language Models (LLMs) into healthcare holds significant potential to enhance diagnostic accuracy and support medical treatment planning. These AI-driven systems can analyze vast datasets, assisting clinicians in identifying diseases, recommending treatments, and predicting patient outcomes. This study evaluates the performance of a range of contemporary LLMs, including both open-source and closed-source models, on the 2024 Portuguese National Exam for medical specialty access (PNA) , a standardized medical knowledge assessment. Our results highlight considerable variation in accuracy and cost-effectiveness, with several models demonstrating performance exceeding human benchmarks for medical students on this specific task. We identify leading models based on a combined score of accuracy and cost, discuss the implications of reasoning methodologies like Chain-of-Thought, and underscore the potential for LLMs to function as valuable complementary tools aiding medical professionals in complex clinical decision-making. Keywords: Artificial Intelligence in Healthcare, Clinical Decision Support, Large Language Models, Medical Diagnosis, Natural Language Processing 1 Introduction Recent advancements in artificial intelligence (AI), particularly Large Language Models (LLMs) such as those developed by Google (Gemini), OpenAI (GPT series), Anthropic (Claude), Meta (LLaMA), and others, have significantly enhanced the potential for supporting medical diagnostics and treatment decision-making. These models, characterized by their ability to process and generate human-like text, show promise in revolutionizing aspects of clinical workflows, patient education, and medical research [ 1 ] . Previous studies have explored the performance of earlier LLMs, such as ChatGPT, on medical licensing exams, including prior editions of the Portuguese National Residency Access Examination (PNA) [ 2 ] , demonstrating promising capabilities but often falling short of top human performance thresholds consistently across all specialties. In many clinical scenarios, particularly those involving complex differential diagnoses or treatment planning, rapid access to and synthesis of relevant information is critical. This study explores the capabilities of current LLMs in the medical domain, focusing on their potential to serve as adjunctive tools for healthcare professionals. The objective is not to replace clinical judgment but to investigate how LLMs can assist doctors by processing complex information, suggesting potential diagnostic pathways, and aiding in treatment considerations. Figure 1 represents this approach. We evaluate a diverse set of both freely available (open-source) and commercially licensed (closed-source) models to understand the performance landscape across different access paradigms, balancing cutting-edge capabilities often found in closed models with the transparency and customizability offered by open-source alternatives. We evaluate models based not only on accuracy but also cost-effectiveness—an essential consideration in healthcare, where budgets and resource allocation can strongly influence the adoption of new technologies. By examining their accuracy on a challenging medical knowledge assessment, we aim to provide insights into their current strengths and limitations as potential aids in the diagnostic and treatment process. Figure 1: Conceptual Diagram 2 The Evolution of LLMs in Healthcare Initially, LLMs faced considerable limitations when dealing with the nuances of medical jargon and the complexity of clinical reasoning. However, contemporary models have made dramatic improvements, driven by greater computational power, larger and more diverse training datasets, and advanced training techniques, including reinforcement learning from human feedback (RLHF) applied to CoT. 2.1 Advances in Foundational Architectures The rapid progression within model families, such as OpenAI’s GPT series (from GPT-3.5 to GPT-4 and beyond), Google’s Gemini models, Anthropic’s Claude series, and Meta’s LLaMA versions, has resulted in substantial improvements in accuracy, contextual understanding, and reasoning capabilities. This is notably demonstrated by the performance of models like GPT-4 on medical licensing assessments [ 3 ] . For instance, on previous editions of the Portuguese PNA (2019-2023), ChatGPT-4o significantly outperformed its predecessor, ChatGPT-3.5, achieving scores sufficient for any medical specialty placement [ 2 ] , foreshadowing the capabilities tested in our current study with even newer models. Figure 2 is a short diagram that represents this evolution. Figure 2: LLMs Beginning and Evolution 2.2 The Current LLM Landscape The field is rapidly evolving, with numerous organizations releasing powerful models. A comparison across various LLMs reveals a wide spectrum of capabilities, architectural differences (e.g., dense vs. Mixture-of-Experts), and access models (open-weight vs. API-only). Understanding the characteristics of these models, such as their knowledge cutoff dates, context window sizes, and licensing terms (as detailed in Table A.1 ), is crucial for evaluating their suitability for specific healthcare applications. Detailed metadata for each evaluated LLM, grouped by developer, is provided in Table A.1 on the following landscape page. 3 Reasoning Methodologies Enhancements Significant advancements in reasoning methodologies are reshaping how LLMs perform complex tasks, including those relevant to diagnostics and therapeutic planning. 3.1 Chain of Thought (CoT) The Chain of Thought (CoT) approach encourages LLMs to generate intermediate reasoning steps before arriving at a final answer, mimicking human cognitive processes. This has been shown to significantly improve performance in logical, mathematical, and complex reasoning tasks [ 4 ] , making it particularly relevant for structured clinical diagnosis scenarios and multi-stage medical reasoning challenges. Several models evaluated in this study incorporate explicit CoT or "thinking" modes (indicated by ⋆ in Table 1 ). Figure 3 represents this mechanism. Figure 3: Chain of Thought 3.2 Chain of Draft (CoD) While Chain of Thought (CoT) significantly improves reasoning by encouraging verbose, step-by-step explanations, this approach often leads to high token usage and increased latency, contrasting with efficient human problem-solving [ 5 ] . Humans typically rely on concise intermediate notes or drafts, capturing only essential information to guide their thinking process. Inspired by this cognitive efficiency, Xu et al. proposed Chain of Draft (CoD) [ 5 ] . This paradigm prompts LLMs to generate minimalistic yet informative intermediate reasoning outputs—essentially concise "drafts"—at each step, rather than fully elaborated explanations. The goal is to focus computational effort on critical insights and calculations necessary to advance towards the solution, stripping away redundant verbosity. Experiments across arithmetic, commonsense, and symbolic reasoning tasks demonstrated that CoD can match or even surpass the accuracy of standard CoT prompting [ 5 ] . Crucially, this performance parity is achieved with a dramatic reduction in resource consumption, using significantly fewer tokens (reportedly as little as 7.6% of CoT tokens in some cases) and thereby lowering both inference cost and latency. This efficiency makes CoD a potentially valuable strategy for deploying LLMs in resource-constrained or time-sensitive environments, such as real-time clinical decision support, where rapid and accurate reasoning is paramount but computational overhead must be minimized. However, the authors note that CoD’s effectiveness, particularly in zero-shot scenarios or with smaller models, can depend on the model’s underlying training and may benefit from few-shot examples or specific fine-tuning [ 5 ] . 3.3 Emerging Diffusion-Based Language Models While the models evaluated in this study predominantly use autoregressive architectures (generating text sequentially token by token), a newer generation of language models based on diffusion processes is emerging. Inspired by the success of diffusion models in image and video generation (e.g., DALL-E, Midjourney, Sora) [ 6 ] , this approach, fundamentally different from sequential generation, is being adapted for text tasks [ 7 ] . Companies like Inception Labs are pioneering commercial-scale diffusion Large Language Models (dLLMs), such as their announced Mercury model [ 8 ] , claiming significant