MedPAIR: Measuring P hysicians and AI R elevance Alignment in Medical Question Answering

Yuexing Hao, Kumail Alhamoud, Hyewon Jeong, Haoran Zhang, Isha Puri, Philip Torr, Mike Schaekermann, Ariel D. Stern, Marzyeh Ghassemi · 2025

Read MedPAIR: Measuring P hysicians and AI R elevance Alignment in Medical Question Answering by Yuexing Hao, Kumail Alhamoud, Hyewon Jeong, Haoran Zhang, Isha Puri, Philip Torr, Mike Schaekermann, Ariel D. Stern, Marzyeh Ghassemi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24040.txt.

Introduction

Yuexing Hao and co-authors build a data set that compares how trainee doctors and AI models judge which facts in a medical case matter, in a paper from 2025. Language models ace many exam-style tests, but a right answer can rest on flawed logic. The set, called MedPAIR, holds 1,300 question pairs. Thirty-six trainees marked each sentence in a case as relevant or not.

The team set those marks beside the marks implied by several models. The models often prized unlike facts from the ones trainees chose. Stripping out the sentences trainees called irrelevant raised scores for both groups. That result cuts both ways: it shows models waste effort on noise, and it gives a cleaner test bed. All the marks are shared in public. The work pushes evaluation past final answers toward sound reasoning.

Catalog record

CREATORYuexing Hao, Kumail Alhamoud, Hyewon Jeong, Haoran Zhang, Isha Puri, Philip Torr, Mike Schaekermann, Ariel D. Stern, Marzyeh Ghassemi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24040

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MedPAIR: Measuring P hysicians and AI R elevance Alignment in Medical Question Answering Yuexing Hao, Kumail Alhamoud, Hyewon Jeong, Haoran Zhang, Isha Puri, Philip Torr, Mike Schaekermann, Ariel D. Stern, Marzyeh Ghassemi Abstract Large Language Models (LLMs) have demonstrated remarkable performance on various medical question-answering (QA) benchmarks, including standardized medical exams. However, correct answers alone do not ensure correct logic, and models may reach accurate conclusions through flawed processes. In this study, we introduce the MedPAIR ( Med ical Dataset Comparing P hysicians and AI R elevance Estimation and Question Answering) dataset to evaluate how physician trainees and LLMs prioritize relevant information when answering QA questions. We obtain annotations on 1,300 QA pairs from 36 physician trainees, labeling each sentence within the question components for relevance. We compare these relevance estimates to those for LLMs, and further evaluate the impact of these “relevant” subsets on downstream task performance for both physician trainees and LLMs. We find that LLMs are frequently not aligned with the content relevance estimates of physician trainees. After filtering out physician trainee-labeled irrelevant sentences, accuracy improves for both the trainees and the LLMs. All LLM and physician trainee-labeled data are available at: http://medpair.csail.mit.edu/ . 1 Introduction Large language models (LLMs) have shown strong performance across a range of medical tasks, with systems like GPT-4 and MedPaLM outperforming human averages on standardized medical examinations cabral_clinical_2024 ; mcduff_towards_2025 . However, many tasks do not reflect the complexity of real-world use cases xia_cares_2024 , and high performance on exam-style datasets may overstate a LLM’s generalizability li_mediq_2024 . In human-facing settings, it is crucial to understand how models filter and prioritize relevant information park_does_2015 . Estimation of contextual relevance is a critical aspect in many applications. Techniques such as semantic entropy farquhar_detecting_2024 , influence functions choe_what_2024 , context attribution cohen-wang_learning_2025 ; liu_attribot_2024 , and evidence inference deyoung_eraser_2020 have been employed to assess which elements within a context hold the most importance. Despite these efforts, existing relevance estimations are often imprecise and noisy, with models sometimes producing misleading or overly confident assessments that deviate from human judgment es_ragas_2024 . Even when estimations appear less noisy, model-generated relevance labels may not concord with those of human experts. This gap is particularly concerning in human-facing domains where alignment with expert judgment is necessary zhao_moverscore_2019 ; schuler_context_2025 . We focus on the question-answering (QA) in clinical contexts, which reflect how physicians synthesize patient data to address specific concerns. Existing medical QA datasets have driven progress in evaluating LLM’s performance on clinically relevant tasks pal_medmcqa_2022 ; zhang_survey_2024 ; jin_rjua-meddqa_2024 ; soni_radqa_2022 . However, QA benchmarks and leader boards primarily assess final answers, providing limited visibility into the underlying rationale toma_clinical_2023 ; adlakha_evaluating_2024 . In this work, we curate a Med ical Dataset comparing P hysician trainees and AI R elevance estimation and question answering - MedPAIR . We design MedPAIR to understand how physician trainees and LLMs select relevant information in structured QA. We collect sentence-level relevance labels on 2000 samples from the four QA benchmark datasets from 36 physician trainees. In parallel, we prompt LLMs to self-report sentence-level relevance choi_identifying_2024 and apply ContextCite cohen-wang_contextcite_2024 , a context attribution framework that maps model outputs to the input sentences most responsible for their generation. This approach allows us to quantify the degree of alignment between human and model assessments of contextual importance. Using these annotations, we evaluate how sentence-level relevance, estimated by either LLMs or humans, affects downstream QA performance. We release the first benchmark and open-source dataset of physician trainee-annotated relevance for patient case QA tasks , enabling direct comparison with LLM-assigned scores. Our full workflow can be found in Figure 1 . 2 Related Work 2.1 Aligning Human and LLM Estimation Previous work has shown that limiting input to relevant information can reduce distraction, streamline evidence integration, and reduce memory requirements khattab_baleen_2021 ; chuang_selfcite_2025 ; lu_learn_2022 ; bucinca_trust_2021 . Ensuring Artificial Intelligence (AI) systems focus on the same input information that physician trainees identify as relevant is crucial to evaluating which clinical details informed each prediction strong_chatbot_2023 ; katz_gpt_2024 . This alignment allows physicians to judge the reliability and explainability of AI suggestions and reduces the risk of mistakes caused by extraneous or misinterpreted inputs chiang_can_2023 . Demonstrating alignment between LLM-selected input context and expert judgment is increasingly recognized as fundamental to earning physician trust in diagnostic AI tools zhou_explainable_2025 ; wu_medreason_2025 ; bansal_does_2021 . Effective clinical decision-making often relies on understanding nuanced input information that can reveal critical insights. In a systematic review, Schuler and colleagues identify 946 distinct contextual factors that influence clinical decisions, demonstrating the complexity of integrating these elements into evidence-based reasoning schuler_context_2025 . For AI systems to be trusted in clinical settings, they must reflect this contextual understanding, prioritizing information in a way that resonates with clinical judgment gaube_non-task_2023 ; hao_retrospective_2025 ; yang_harnessing_2023 . Transparent alignment between AI reasoning and clinician perspectives can reduce the risks of misleading correlations and enhance trust in AI-based clinical decision support chanda_dermatologist-like_2024 ; rong_towards_2024 , where alignment improved confidence in AI-assisted diagnoses. Aligning AI models with physicians’ nuanced contextual understanding is essential for their acceptance by the medical establishment and effective integration into clinical practice. 2.2 Challenges in Comparing LLM and Human Relevance Judgments Recent work has questioned the reliability of LLMs in consistently judging the relevance of information chen_reasoning_nodate . Although models such as GPT-4 demonstrate high average performance across benchmark datasets, studies have documented significant variance in self-reported labels when identical prompts are issued multiple times cohen-wang_contextcite_2024 ; gao_enabling_2023 ; funkquist_citebench_2023 . This inconsistency is attributed to several factors: prompt sensitivity wang_prompt_2024 ; razavi_benchmarking_2025 ; savage_diagnostic_2024 , stochastic decoding procedures, architectural idiosyncrasies of the model, and ambiguity in input data. Even in deterministic settings (e.g., temperature zero), LLMs can produce divergent responses due to underlying randomness or unstable decision boundaries ahn_prompt-reverse_2025 . Empirical evaluations confirm that model agreement across repeated prompts is rarely perfect, with accuracy fluctuations of up to 10% depending on task complexity and phrasing atil_llm_2024 . Further highlighting this gap, recent studies report that between 50% and 90% of LLM-generated medical answers are not fully supported by the cited references wu_automated_2025 . There are multiple ways to evaluate LLM’s relevance judgments in the input context, for example semantic entropy using probabilistic approaches to detect hallucination farquhar_detecting_2024 . These challenges are particularly pronounced in clinical contexts; widely used medical QA benchmarks provide limited visibility into how models interpret context. For example, PubMedQA jin_pubmedqa_2019 does not offer detailed annotations identifying which parts of the text are crucial for answering the question singhal_toward_2025 . Similarly, MedQA jin_what_2021 lacks expert-provided rationales or sentence-level relevance labels. Other datasets, including MedMCQA pal_medmcqa_2022 , MMLU’s medical subsets hendrycks_measuring_2021 , MetaMedQA griot_large_2025 , and MEDIQ li_mediq_2024 , also prioritize answer correctness. 3 The MedPAIR Dataset 3.1 QA Dataset Setup To examine the alignment between physician trainees’ and LLMs’ assessments of relevance, we deliberately concentrate on existing QA pairs grounded in specific patient case scenarios, rather than general evidence-based questions. This focus is intended to better simulate authentic clinical contexts. Therefore, we draw on our four datasets Massive Multitask Language Understanding (MMLU)-precision medicine (272 QAs) hendrycks_measuring_2021 , Medbullets (298 QAs), JAMA Clinical Challenge dataset (1,034 QAs) chen_benchmarking_2025 ,

Read the complete text