Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries

Jean Feng, Vishal Patel, Patrick Heagerty, Yifan Mai, Venkatesh Sivaraman, Patrick Vossler, Jialin Ouyang, Anupam B. Jena · 2026

Read Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries by Jean Feng, Vishal Patel, Patrick Heagerty, Yifan Mai, Venkatesh Sivaraman, Patrick Vossler, Jialin Ouyang, Anupam B. Jena (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28960.txt.

Introduction

Feng and colleagues test clinical AI tools on the questions doctors actually ask at the point of care, rather than exam-style problems. Their blinded evaluation uses 620 real queries submitted by physicians across thirty specialties to the OpenEvidence platform, plus 187 questions from a public benchmark. In total, 149 practising physicians from 36 states compared answers head-to-head, each matched to the question's specialty.

Four systems competed: three frontier general-purpose models and one specialised clinical tool. Graders scored clinical utility, accuracy, source quality, verifiability and completeness. Systems specialised tool scored highest on all five dimensions, with win margins of 25 to 39 percentage points on the real queries. Results held across checks varying citation display, answer length and user status. AI judges differed systematically from the physicians, though both picked the same best model. The team release the query set as a public benchmark.

Catalog record

CREATORJean Feng, Vishal Patel, Patrick Heagerty, Yifan Mai, Venkatesh Sivaraman, Patrick Vossler, Jialin Ouyang, Anupam B. Jena
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.28960

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries Jean Feng, Vishal Patel, Patrick Heagerty, Yifan Mai, Venkatesh Sivaraman, Patrick Vossler, Jialin Ouyang, Anupam B. Jena Abstract Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in practice. We report a blinded evaluation built on 620 Real-world Point-Of-Care Queries (Real-POCQi) submitted to the OpenEvidence (OE) platform by physicians spanning 30 specialties, as well as 187 questions from HealthBench. 149 practicing physicians across 36 states made head-to-head comparisons between answers from three frontier general-purpose models (Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5) and a specialized clinical tool (OE), with graders matched to each question’s specialty. When comparing answers along five dimensions relevant to clinical decision support—accuracy, clinical utility, source quality, verifiability, and completeness—physicians scored the specialized tool highest on all axes; in the primary analysis on Real-POCQi, win differences (margins between win and loss rates) ranged from 25 to 39 percentage points ( p < < 0.001). Results remained consistent in sensitivity analyses stratifying by citation display, answer length, OE-user status, and Real-POCQi versus HealthBench. In parallel, LLM judges were found to systematically differ from expert judges, though both generally agreed on the best model. These findings underscore two conclusions. First, AI tool evaluations should reflect real-world query distributions and use expert judges that mirror the specialization defining modern medicine. Second, the consistent advantage of the specialized tool over general-purpose models does not mean that the latter cannot serve similar purposes, but that targeted engineering and customization can yield meaningful gains in performance and utility for its users. We release Real-POCQi as a public benchmark for further analysis, as well as the prespecified statistical analysis for reproducing results of this study. 1 Introduction Clinicians are among the rapid adopters of AI tools, with tens of millions of clinical requests made each month by U.S. physicians ( 22 ) . Given the stakes of the decisions these tools inform, and the speed at which their capabilities and adoption are expanding, rigorous and safety-focused evaluation is essential. This imperative has spurred a proliferation of research evaluating medical AI models ( 17 ; 2 ; 3 ; 28 ) , but building evaluations that faithfully reflect real clinical practice remains a fundamental challenge ( 1 ) . Medical AI evaluation has largely relied on standardized benchmarks — fixed, shared datasets of test questions with predetermined correct answers or grading rubrics, such as those drawn from licensing examinations. There is growing concern that these do not reflect the real-world challenges physicians face when using these models in practice, due to fundamental limitations in both how evaluation data are collected and how answers are graded. First, public datasets rarely capture the real-world distribution of questions physicians ask during care delivery, in part because such queries are difficult to collect at scale and are seldom released by the proprietary platforms through which they are posed. Early benchmarks relied heavily on multiple-choice items drawn from standardized licensing examinations ( 17 ; 25 ) , while more recent efforts use questions co-designed by physicians and AI agents ( 2 ) . In both cases, the questions are hypothetical by default, tend to oversimplify the clinical situation, and reflect speculative rather than empirically grounded assumptions regarding questions asked at the point of care. Second, due to the difficulty in obtaining high-quality annotations in medicine, grading of AI on clinical topics typically rely on a small number of physicians to rate questions across many domains, disregarding the specialization that defines modern medicine. Furthermore, fixed rubrics are commonly used, on the supposition that this grading scheme is more cost-efficient ( 28 ) . However, recent works have shown that rubrics are substantially less accurate than conducting head-to-head comparisons, particularly in high-expertise domains ( 31 ) , because rubrics not only presuppose what makes an answer good but also make blanket assumptions. This does not reflect the real-world needs of practicing physicians at the point of care. At the same time, there is now a debate regarding whether one can forgo human experts altogether by using a LLM judge or jury instead ( 7 ; 3 ; 29 ) . For AI-based clinical decision support systems, the key question is whether language models can reliably arbitrate clinical reasoning for questions that arise at the point-of-care, which has been largely untested up to now. Given these gaps, there is a need to move beyond standardized benchmarks toward evaluation grounded in the realities of clinical practice, built on the questions clinicians ask and the judgment of appropriately matched experts. Figure 1: Study design and composition of the benchmark dataset. Specialty-matched, licensed and actively practicing U.S. physicians evaluated four chat-based medical AI systems — GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.8 and OpenEvidence (OE) — on real-world clinical decision support questions in a randomized, blinded, pairwise design. a , Study schematic. The question bank (Real-POCQi) was derived from deidentified and rewritten OE real user queries, sampled across 36 specialties (IQVIA database) and filtered per exclusion criteria; HealthBench questions were additionally included for sensitivity analysis. Each evaluator was assigned a single render mode held fixed throughout — text-only (primary) or text with citations (secondary); for every question a random pair of the four systems was drawn, and the two answers were shown in randomized A/B positions with vendor identities hidden and rendered identically. Each comparison was rated independently on five dimensions (accuracy, clinical utility, source quality, verifiability and completeness) on a five-point preference scale from A much better to B much better. b , Prevalence of each specialty area among sampled and rated questions in the dataset. c , Distribution of question types across the dataset, as identified through LLM-based clustering. Here we report a large, blinded evaluation built on Real-world Point-Of-Care Queries (Real-POCQi) submitted by physicians to the OpenEvidence (OE) platform, with a sensitivity analysis on questions from the HealthBench benchmark ( 2 ) (Figure 1 ). We publicly release the questions, answers, ratings, and analysis code for Real-POCQi ( https://huggingface.co/datasets/jjfenglab/Real-POCQi , https://github.com/jjfenglab/Real-POCQi-statistics ). Questions in Real-POCQi were drawn from the more than one million questions submitted to OE each day by physicians across the U.S., spanning topics from diagnosis and prognosis to treatment selection and drug safety, and specialties from generalist domains such as primary care to subspecialties such as nephrology. In a survey sent to randomly selected physicians across the U.S., 149 respondents across 36 states were matched to questions by clinical topic and made blinded, head-to-head comparisons between answers from four AI models: three frontier general-purpose models (Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5) and one specialized clinical tool (OE). Through the pairwise comparison approach popularized by LLM arenas ( 8 ) , answers by AI systems were assessed along five dimensions related to point-of-care clinical decisions (accuracy, clinical utility, source quality, verifiability, and completeness). The final Real-POCQi dataset constitutes the largest specialty-matched public benchmark of real-world clinical queries asked by practicing clinicians, with 620 questions, 1156 comparisons, and 30 specialties. With it, we evaluate the ability of general-purpose and specialized AI models to generate answers that reflect the needs of

Read the complete text