HealthBench: Evaluating Large Language Models Towards Improved Human Health
Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, Karan Singhal · 2025
Read HealthBench: Evaluating Large Language Models Towards Improved Human Health by Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, Karan Singhal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08775.txt.
Introduction
Rahul Arora and co-authors build a health test for language models in this 2025 paper. HealthBench grades models on real-style medical talks rather than quiz questions. It holds 5,000 multi-turn chats between a model and a user or a clinician.
Doctors wrote the grading guides: 262 physicians shaped over 48,000 checks. The checks span emergencies, clinical data work, and global health, and they score accuracy and conduct. Open-ended replies make the test closer to real use than multiple choice. The benchmark is open-source. It gives health teams a shared way to judge model safety and skill.
Catalog record
| CREATOR | Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, Karan Singhal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08775
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HealthBench: Evaluating Large Language Models Towards Improved Human Health Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, Karan Singhal Abstract We present HealthBench , an open-source benchmark measuring the performance and safety of large language models in healthcare. HealthBench consists of 5,000 multi-turn conversations between a model and an individual user or healthcare professional. Responses are evaluated using conversation-specific rubrics created by 262 physicians. Unlike previous multiple-choice or short-answer benchmarks, HealthBench enables realistic, open-ended evaluation through 48,562 unique rubric criteria spanning several health contexts (e.g., emergencies, transforming clinical data, global health) and behavioral dimensions (e.g., accuracy, instruction following, communication). HealthBench performance over the last two years reflects steady initial progress (compare GPT-3.5 Turbo’s 16 % 16\% to GPT-4o’s 32 % 32\% ) and more rapid recent improvements (o3 scores 60 % 60\% ). Smaller models have especially improved: GPT-4.1 nano outperforms GPT-4o and is 25 times cheaper. We additionally release two HealthBench variations: HealthBench Consensus, which includes 34 34 particularly important dimensions of model behavior validated via physician consensus, and HealthBench Hard, where the current top score is 32 % 32\% . We hope that HealthBench grounds progress towards model development and applications that benefit human health. 1 1 1 HealthBench can be found at https://github.com/openai/simple-evals . 1 Introduction Artificial intelligence (AI) systems are increasingly used in health, offering the potential to expand access to health information, support clinicians in delivering high-quality care, and help people make better health decisions ( Esteva et al., 2017 , Gulshan et al., 2016 , Beam and Kohane, 2018 , Topol, 2019 ) . In particular, large language models (LLMs) can encode vast clinical knowledge, reason over complex inputs, and adapt flexibly to user needs ( Singhal et al., 2023 , Singhal et al., 2025 , Nori et al., 2023 ) . If harnessed appropriately and deployed effectively, LLMs could ameliorate persistent gaps in information access, care quality, and outcomes, especially in low-resource settings. Realizing the potential of AI in healthcare requires that these models perform well and behave safely in diverse and high-stakes health situations. Evaluations are essential as shared standards and markers of progress for the field, yet existing health evaluations have significant limitations. These evaluations often fall short on three key dimensions: 1. Meaningful : Do scores reflect real-world impact? Many existing evaluations rely on multiple-choice exams or narrow clinical questions, making them poorly aligned with the open-ended, dynamic nature of real situations and workflows for healthcare professionals and individual users. 2. Trustworthy : Are scores faithful indicators of physician judgment? Many existing evaluations lack validation against expert medical opinions. 3. Unsaturated : Do benchmarks still have headroom to support progress? Many existing evaluations offer no room for improvement, failing to incentivize model developers to improve performance. Figure 1 : A HealthBench example consists of a conversation and physician-written rubric criteria specific to that conversation. A model-based grader scores the response against each criterion. Figure 2 : HealthBench scores of OpenAI models compared with inference cost per example. In this work, we present HealthBench , an open-source benchmark designed to address these gaps. HealthBench was developed in partnership with 262 physicians who collectively have practiced in 60 countries. HealthBench comprises 5,000 realistic health conversations between a model and an individual user or healthcare professional. Conversations were selected for relevance, realism, and difficulty, and span a wide range of geographies, languages, and healthcare personas. Given a conversation, the task for an LLM is to respond to the last user message. HealthBench is a rubric evaluation . To grade open-ended model responses, we score them against a conversation-specific physician-written rubric, 2 2 2 An “example” in this paper refers to an individual conversation between an LLM and a layperson or a healthcare professional. composed of self-contained, objective criteria. Criteria capture attributes that a response should be rewarded or penalized for in the context of that conversation (e.g., specific facts that should be included, aspects of clear communication, common misconceptions about a topic) and their relative importance. HealthBench evaluates 48,562 unique criteria across all conversations. A model-based grader, validated against physician judgment, is used to score any given response against the rubric criteria for that conversation. In addition to a single-number overall score, HealthBench also provides targeted measurement of health behavior. Model performance can be stratified by both themes , which are high-level categories of health-related tasks that reflect distinct challenges in real-world health interactions (e.g., emergency referrals, global health, or seeking context), and axes , which define the specific behavioral dimensions that each rubric criterion evaluates, such as clinical accuracy, communication quality, and context awareness. This helps diagnose the specific behaviors of different AI models and indicates which types of conversations and performance dimensions have room for improvement. Using HealthBench, we evaluate a range of state-of-the-art LLMs. Recent models have improved rapidly on HealthBench, across frontier performance, cost (shown in Fig. 2 ), and reliability. We establish human baselines where physicians were instructed to produce responses both with and without model assistance, finding that recent models produce higher quality responses than physicians unless physicians are assisted by the same models. Substantial room for model improvement remains, particularly in worst-case performance and context-seeking behavior. HealthBench is a meaningful, trustworthy, and unsaturated benchmark for AI systems in health, offering a comprehensive and extensible standard for researchers aiming to develop safe and beneficial models. We are releasing HealthBench openly to ground progress, foster collaboration, and support the broader goal of ensuring that AI advances translate into meaningful improvements in human health. An overview of our contributions is as follows: • HealthBench includes 5,000 realistic conversations between models and users (including individual users and healthcare professionals) across seven themes and five axes, measuring 48,562 unique aspects of model behavior via rubric evaluation ( Section 2 ). • HealthBench was produced with 262 physicians across 26 specialties and with practice experience in 60 countries ( Section 4.1 ). • We measure HealthBench performance across frontier models. We find that recent models have improved rapidly, across frontier performance, performance at cost, and reliability ( Section 6 ). • As a baseline, we asked physicians to write responses to HealthBench examples with and without reference responses from models, and found that models outperformed the unassisted expert baseline, and physicians could improve responses for models from September 2024 but not April 2025 ( Section 7 ). • We introduce and report results for two variations of HealthBench: HealthBench Consensus, which measures 34 particularly important aspects of model behavior and has been validated by the consensus of multiple physicians, and HealthBench Hard, on which no model we evaluated scored above 32\% ( Section 3 ). • To understand trustworthiness of HealthBench, we measure agreement between model and physician grading, and find that model-physician agreement is similar to physician-physician agreement ( Section 8 ). • We release HealthBench data and code via OpenAI’s simple-evals repository ( Section 12 ). 2 Rubric evaluations HealthBench is a rubric evaluation ( Starace et al., 2025 , Lin et al., 2025 , Sirdeshmukh et al., 2025 , Scale AI, 2025 , Fast et al., 2024 ) , which means that a