MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMs

Zhan Qu, Michael Färber · 2025

Read MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMs by Zhan Qu, Michael Färber (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20822.txt.

Introduction

Zhan Qu and Michael Färber built MediEval to test medical language models on knowledge and patient context together. Older tests either quizzed textbook facts alone or judged bedside reasoning without checking correctness. MediEval instead links real patient records from MIMIC-IV to a knowledge base drawn from standard biomedical vocabularies.

The benchmark writes true and deliberately false medical statements inside real patient stories, then grades models on grounding and consistency together. Testing exposed dangerous failure modes, including invented support and answers that flip the truth. The authors' fine-tuning fix, CoRFu, penalizes unsafe confusions more heavily. It raised accuracy sharply and removed the truth-flipping errors entirely in their tests.

Catalog record

CREATORZhan Qu, Michael Färber
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20822

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMs Zhan Qu, Michael Färber Abstract Large Language Models (LLMs) are increasingly applied to medicine, yet their adoption is limited by concerns over reliability and safety. Existing evaluations either test factual medical knowledge in isolation or assess patient-level reasoning without verifying correctness, leaving a critical gap. We introduce MediEval, a benchmark that links MIMIC-IV electronic health records (EHRs) to a unified knowledge base built from UMLS and other biomedical vocabularies. MediEval generates diverse factual and counterfactual medical statements within real patient contexts, enabling systematic evaluation across a 4-quadrant framework that jointly considers knowledge grounding and contextual consistency. Using this framework, we identify critical failure modes, including hallucinated support and truth inversion, that current proprietary, open-source, and domain-specific LLMs frequently exhibit. To address these risks, we propose Counterfactual Risk-Aware Fine-tuning (CoRFu), a DPO-based method with an asymmetric penalty targeting unsafe confusions. CoRFu improves by +16.4 macro-F1 points over the base model and eliminates truth inversion errors, demonstrating both higher accuracy and substantially greater safety. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, with medicine being among the most high-impact areas of application. In clinical contexts, LLMs have been explored for tasks such as summarizing electronic health records (EHRs), generating discharge instructions, providing clinical decision support, and answering medical questions ( 18 ; 21 ; 13 ; 20 ; 16 ) . Their appeal lies in the ability to integrate unstructured text with medical knowledge, potentially reducing documentation burden and assisting clinicians in decision-making. Translating research prototypes into real-world deployment hinges critically on reliability and safety. Unlike generic NLP tasks, medical reasoning requires not only factual correctness but also contextual grounding in patient-specific data while adhering to verified medical knowledge. Errors in this setting are not mere degradations in performance but risks that can directly translate into patient harm ( 25 ; 31 ; 8 ) . A critical challenge is that LLMs often fail to apply medical knowledge consistently within the heterogeneous and noisy context of patient records ( 35 ) . For example, a model may state that metformin, a therapy for type 2 diabetes, is contraindicated in severe renal impairment, yet fail to apply this knowledge when the condition appears in a noisy and heterogeneous patient record. The cause of such errors may be that current models are trained to recall facts in isolation rather than to integrate them with diverse patient information. Such inconsistencies expose a critical gap between knowing medical facts and using them safely. Figure 1: Overview of the current work with a real example; texts in blue indicate the extracted sample. Existing evaluation paradigms only partially address this gap. Medical data are uniquely challenging because patient records are heterogeneous with free-text clinical notes and tabular entries coded in different systems for diagnoses, procedures, and medications. Medical knowledge is hierarchical and ontology-driven (e.g., UMLS, SNOMED CT, RxNorm), but its large scale, noise, and limited cross-vocabulary connectivity make consistent reasoning difficult. Benchmarks based on EHRs assess the ability to extract or reason over structured data, but often reduce the task to retrieval or serialization without verifying medical soundness ( 12 ) . In contrast, knowledge-based evaluations test whether LLMs can handle logical transformations of medical facts ( 34 ; 23 ) , but do not connect reasoning to real patient contexts. The field thus lacks a unified framework that probes whether LLMs can (i) remain faithful to medical knowledge and (ii) apply it consistently to individual patient records. In this paper, we address this gap with MediEval (Figure 1 ), a benchmark and evaluation framework that links real patient records (MIMIC-IV) ( 10 ) with a unified biomedical knowledge base built from UMLS ( 2 ) , SNOMED CT ( 5 ) , and RxNorm ( 11 ; 14 ) . To ensure rigorous construction, MediEval constructs evaluation statements by applying graph-guided substitutions and recombinations within biomedical ontologies, with plausibility checks to ensure that true cases remain clinically valid while false ones are realistic and challenging. These statements are embedded in real patient contexts, enabling evaluation of quadrant-level reasoning, defined as True/False with respect to medical knowledge and Supported/Unsupported with respect to the patient record. To capture safety-critical errors, we introduce two new metrics: Hallucinated Support Rate (HSR) and Truth Inversion Rate (TIR). We then conduct a comprehensive evaluation of proprietary, open-source (general-purposes), and biomedical LLMs under this unified protocol, revealing consistent weaknesses across quadrants. To mitigate the identified risks, we introduce Counterfactual Risk-Aware Fine-tuning (CoRFu) , a DPO-based method that uses quadrant-structured preference pairs and an asymmetric penalty to push models to prefer “True+Supported” responses over safety-critical confusions (e.g., “False+Supported”). This risk-aware, counterfactual approach targets precisely the failure modes that conventional preference optimization overlooks. Our code and data are publicly available 1 1 1 https://anonymous.4open.science/r/MediEval-5FA5/ . Our key contributions are: • MediEval Benchmark. A unified benchmark linking EHRs with biomedical ontologies to test both factual grounding and patient consistency, with novel safety evaluation metrics. • Comprehensive Evaluation. Cross-model evaluation of proprietary, open-source, and biomedical LLMs under identical settings, revealing systematic quadrant-level weaknesses and safety-critical error patterns. • Counterfactual Risk-Aware Fine-Tuning (CoRFu). A risk-aware extension of DPO with asymmetric penalties and quadrant-aware approach that reduces safety-critical errors in medical reasoning tasks. 2 Related Work A growing body of work investigates whether LLMs can handle complex information in EHRs. One study focused on data serialization and retrieval, with performance sensitive to prompt design and feature selection ( 12 ) . Another work has explored long-context modeling, showing some architectures can process entire patient timelines with > 10k events and achieve greater robustness to irregular temporal patterns ( 27 ) . Other approaches integrate structured and unstructured data, for instance, by using small auxiliary models as “knowledge triggers” to support tabular prediction ( 29 ) or by designing code-aware representations that capture semantic and temporal structure in diagnosis prediction ( 24 ) . These studies demonstrate progress in surfacing and encoding patient information, but whether model outputs are consistent with established clinical knowledge remains underexplored. Another line of research investigates the reliability of the medical knowledge encoded in LLMs. Dynamic probing methods show that models often achieve low joint accuracy when facts are rephrased ( 34 ) , while multifaceted evaluations reveal brittle performance on tasks requiring comparison, verification, or rectification ( 33 ) . Broader studies also report steep declines when moving from factual recall to scenario-based reasoning as cognitive complexity increases ( 35 ) . However, even when scenarios are considered, these evaluations do not disentangle whether errors arise from factual misunderstanding, failures of contextual grounding, or both. Several approaches have sought to improve the factual reliability of LLMs by grounding them in structured biomedical knowledge. These include triplet generation with knowledge graph verification ( 22 ) , alignment with biomedical knowledge graph embeddings ( 19 ) , and graph-based retrieval augmentation for medical question-answering ( 28 ) . Scholars have also critiqued the evaluation paradigms: some argue that multiple-choice questions reward shallow pattern recognition rather than genuine medical knowledge ( 7 ) , while others stress that medical benchmarks must prioritize construct validity to remain clinically meaningful ( 1 )

Read the complete text