OpenAI’s HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogério Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam · 2025

Read OpenAI’s HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries by Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogério Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02594.txt.

Introduction

Sandhanakrishnan Ravichandran and seven co-authors present a 2025 medical AI evaluation of their clinical assistant DR. INFO on HealthBench, a benchmark of open-ended health conversations graded by expert rubrics. Multiple-choice tests miss context handling and uncertainty, which matter in real care.

The assistant uses an agentic retrieval design, searching sources before answering. On the benchmark's hard subset of a thousand cases, it scored well above leading frontier models named in the paper, across accuracy, completeness, and instruction following. A smaller head-to-head test against two rival assistants kept the lead. Weak spots remained in context awareness and completeness. The authors argue that behaviour-level, rubric-based grading is the right way to judge clinical support systems.

Catalog record

CREATORSandhanakrishnan Ravichandran, Shivesh Kumar, Rogério Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02594

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OpenAI’s HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogério Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam Abstract Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stake clincal scenarios. Traditional evaluations are often limited to multiple-choice questions that fail to capture essential competencies such as contextual reasoning, awareness and uncertainty handling etc. To address these limitations, we evaluate our agentic, RAG-based clinical support assistant, DR.INFO , using HealthBench , a rubric-driven benchmark composed of open-ended, expert-annotated health conversations. On the Hard subset of 1,000 challenging examples, DR.INFO achieves a HealthBench score of 0.51 , substantially outperforming leading frontier LLMs (GPT-5, o3, Grok 3, GPT-4, Gemini 2.5, etc.) across all behavioral axes (accuracy, completeness, instruction following, etc.). In a separate 100-sample evaluation against similar agentic RAG assistants (OpenEvidence, Pathway.md), it maintains a performance lead with a health-bench score of 0.54 . These results highlight DR.INFO ’s strengths in communication, instruction following, and accuracy, while also revealing areas for improvement in context awareness and completeness of a response. Overall, the findings underscore the utility of behavior-level, rubric-based evaluation for building a reliable and trustworthy AI-enabled clinical support assistant. 1 Introduction Large Language Models (LLMs) are increasingly being adopted in healthcare to improve access to information, enable efficient summarization and retrieval from large medical corpora, support clinical decision-making, and ultimately enhance patient outcomes [ 1 ] [ 2 ] . These models demonstrate particular promise due to their ability to encode extensive biomedical knowledge, interpret complex natural language queries, and generate tailored, context-aware responses [ 3 ] [ 4 ] . Notably, a recent real-world deployment (“AI Consult”) by OpenAI and Penda Health reported 16% fewer diagnostic errors and 13% fewer treatment errors when clinicians used the LLM-based tool, further underscoring this potential [ 5 ] . Several frameworks have been used to evaluate medical LLM-based systems. The United States Medical Licensing Examination (USMLE) is one of the most widely used, often serving as a proxy for foundational medical knowledge assessment [ 6 ] . Other evaluations include the AMIE benchmark, designed for differential diagnosis tasks, and R‑IDEA scoring, which compares clinical reasoning patterns between LLMs and human physicians [ 7 , 8 ] . Among multiple-choice style benchmarks, MedQA [ 9 ] consists of board-style USMLE questions designed to test factual and clinical knowledge; MedMCQA [ 10 ] offers over 194,000 questions across 21 specialties sourced from Indian medical entrance exams; and PubMedQA [ 11 ] includes yes/no/maybe biomedical questions requiring reasoning over PubMed abstracts. These MCQ datasets provide valuable insight into medical knowledge retrieval, though they do not fully capture the interactive and behavioral demands of real-world clinical use. However, real-world clinical deployment demands more than strong performance on academic or recall-based benchmarks. Evaluating an LLM’s readiness for healthcare settings requires understanding not only what it knows, but how it communicates, reasons under uncertainty, and prioritizes patient safety. Traditional evaluation methods such as the USMLE fall short in this regard. While benchmarks like USMLE remain widely used to assess foundational medical knowledge, they primarily test factual recall and clinical reasoning within constrained multiple-choice formats. These settings do not capture the open-ended, conversational, and high-stakes nature of real-world healthcare interactions, nor do they measure essential dimensions such as contextual awareness, safety behavior, and communication clarity. As highlighted by Nori et al., “by reducing the sequential diagnosis cycle to a one-turn multiple-choice quiz, static benchmarks risk overstating model competence and obscure potential weaknesses including premature diagnostic closure, indiscriminate test ordering, and anchoring on early hypotheses” [ 12 ] . To better understand the strengths and limitations of LLMs in real-world clinical contexts, OpenAI introduced HealthBench [ 13 ] , a comprehensive benchmark comprising 5,000 realistic, open-ended conversations between large language models and users, including both laypeople and healthcare professionals. Each conversation is evaluated using a physician-authored rubric with fine-grained, clinically relevant criteria spanning multiple behavioral axes such as accuracy, communication, context seeking, hedging, and completeness. This multi-dimensional evaluation framework offers a much richer, safety-oriented lens on model performance than conventional benchmarks. At Synduct , we developed an agentic retrieval-augmented generation (RAG)-based clinical support assistant, DR.INFO [ 14 ] , which achieved a 95.4% accuracy on the USMLE benchmark [ 6 ] . DR.INFO is built based on recent advances in medical RAG architectures that demonstrate both high accuracy and clinical reliability [ 15 ] [ 16 ] . While this result reflects solid foundational knowledge and reasoning capability, we hypothesized that standardized exams like the USMLE may not sufficiently capture the practical competencies or potential failure modes of DR.INFO in complex, real-world clinical scenarios. To test this, we evaluated DR.INFO using HealthBench [ 13 ] dataset. HealthBench consists of a Consensus subset, which features queries validated for expert agreement and reliability, and the Hard subset, comprising particularly challenging, high-stakes interactions including emergency scenarios. Our aim was to assess not only what DR.INFO knows, but also how it behaves especially under uncertainty and across diverse global contexts. Specifically, HealthBench enables a multi-dimensional evaluation of key clinical behaviors, including the completeness of a response, clinical accuracy, hedging scenarios, adherence to user instructions, context-seeking behavior, and clarity of communication [ 13 ] . This allows us to measure DR.INFO ’s performance not just as a knowledge engine, but as a clinically aligned assistant capable of reasoning through complex, real-world medical conversations. This article presents a comprehensive evaluation of DR.INFO using HealthBench’s structured, rubric-based framework. We benchmark its performance not only against specialized clinical LLM systems such as OpenEvidence and Pathway.md, but also against leading general-purpose models including GPT-4o, Gemini 2.5 Pro, and Claude Opus. Through these comparisons, we aim to characterize the current frontier of clinically capable LLMs and to identify the strengths and limitations of retrieval‑augmented, agentic architectures in addressing the complex demands of real-world medical communication and reasoning. The evaluation also provides a systematic lens through which to identify areas for internal improvement, helping guide the continued development of DR.INFO toward safe, trustworthy, and effective clinical deployment. 2 Dataset Overview HealthBench is a large-scale, physician-curated benchmark comprising 5,000 clinically realistic examples designed to evaluate the performance of large language models in realistic healthcare scenarios. Each example contains a conversation and a set of rubric criteria tailored to that conversation [ 13 ] . Conversations in HealthBench may be either: • Single-turn: A single user message to which the model must respond. • Multi-turn: Alternating user and model messages, ending with a user prompt. 2.1 HealthBench Consensus Subset HealthBench Consensus [ 13 ] is a filtered subset of the dataset, containing only those 3,671 examples that include one or more positive consensus criteria. This subset is particularly useful for studying high-confidence failure cases and for validating model behavior against physician consensus. 2.2 HealthBench Hard Subset HealthBench Hard [ 13 ] is a challenge subset consisting of 1,000 examples identified as especially difficult for current frontier models. It includes both positive and negative scoring criteria and captures cases with high clinical complexity and ambiguity. In our

Read the complete text