EHRStruct: A Comprehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Tasks
Xiao Yang, Xuejiao Zhao, Zhiqi Shen · 2025
Read EHRStruct: A Comprehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Tasks by Xiao Yang, Xuejiao Zhao, Zhiqi Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08206.txt.
Introduction
Xiao Yang, Xuejiao Zhao, and Zhiqi Shen introduce EHRStruct, a benchmark for testing language models on structured electronic health record data. Patient records stored in relational tables guide clinical decisions, and models are increasingly asked to reason over them. Without standard tasks and scoring, claims about model skill in this setting cannot be compared fairly.
EHRStruct defines eleven tasks with twenty two hundred samples drawn from two widely used record data sets. Twenty general and medical models are evaluated, along with factors such as input format, few shot use, and fine tuning. The authors also propose EHRMaster, a code augmented method that reaches top performance on the benchmark.
Catalog record
| CREATOR | Xiao Yang, Xuejiao Zhao, Zhiqi Shen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08206
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EHRStruct: A Comprehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Tasks Xiao Yang, Xuejiao Zhao, Zhiqi Shen Abstract Structured Electronic Health Record (EHR) data stores patient information in relational tables and plays a central role in clinical decision-making. Recent advances have explored the use of large language models (LLMs) to process such data, showing promise across various clinical tasks. However, the absence of standardized evaluation frameworks and clearly defined tasks makes it difficult to systematically assess and compare LLM performance on structured EHR data. To address these evaluation challenges, we introduce EHRStruct, a benchmark specifically designed to evaluate LLMs on structured EHR tasks. EHRStruct defines 11 representative tasks spanning diverse clinical needs and includes 2,200 task-specific evaluation samples derived from two widely used EHR datasets. We use EHRStruct to evaluate 20 advanced and representative LLMs, covering both general and medical models. We further analyze key factors influencing model performance, including input formats, few-shot generalisation, and finetuning strategies, and compare results with 11 state-of-the-art LLM-based enhancement methods for structured data reasoning. Our results indicate that many structured EHR tasks place high demands on the understanding and reasoning capabilities of LLMs. In response, we propose EHRMaster, a code-augmented method that achieves state-of-the-art performance and offers practical insights to guide future research. Code — https://github.com/YXNTU/EHRStruct Project page — https://yxntu.github.io/proEHRStruct/ 1 Introduction Structured electronic health records (EHRs) ( 13 ) store patient information in relational tables, including diagnoses, medications, and laboratory results. Each entry in records corresponds to a specific clinical event or measurement and is often timestamped to capture longitudinal patient trajectories. In contrast to traditional methods such as SQL-based querying, large language models (LLMs) ( 2 ) offer greater flexibility, more powerful reasoning abilities, and a natural language interface, and have been increasingly adopted for structured EHR modeling ( 18 ) . The application of LLMs to structured EHR data has recently attracted growing attention, emerging as a promising frontier for clinical AI research. However, effectively leveraging such data remains challenging even for state-of-the-art LLMs, due to the need for tabular understanding, clinical reasoning, and alignment with user intent ( 30 ) . Recent efforts have attempted to tackle these aspects from different angles: Yang et al. ( 43 ) tackle tabular understanding by automating the generation of standardized summary tables from structured clinical trial data. Zhu et al. ( 49 ) focus on clinical reasoning tasks, developing a prompting-based approach for predicting outcomes such as mortality, length of stay, and readmission from longitudinal EHR records. Kwon et al. ( 17 ) address alignment with user intent by verifying the semantic consistency between structured EHR tables and unstructured clinical notes. Despite promising advances, existing work lacks a unified evaluation framework for assessing LLMs on structured EHR tasks ( 24 ) . This issue manifests in several ways. First, most studies focus on a limited set of tasks—such as disease prediction ( 7 ; 15 ; 14 ) , mortality risk estimation ( 40 ) , information extraction ( 41 ; 16 ) , and arithmetic reasoning over tabular data ( 43 ) —while leaving many clinically important use cases underexplored, including medication recommendation ( 35 ; 31 ) and clinical named entity recognition ( 25 ; 20 ) . Second, even when addressing the same task (e.g., disease prediction), prior studies ( 49 ; 7 ; 15 ; 14 ) often use different datasets and evaluation protocols, limiting reproducibility and hindering fair model comparison. Third, there is no consensus on input formatting strategies or experimental setups, leading to inconsistencies in how structured data are presented to LLMs. Fourth, existing evaluation metrics provide limited interpretability, making it difficult to understand which specific reasoning capabilities contribute to model successes or failures. Task Scenarios Task Levels Task Categories Task IDs Metrics Data-Driven Understanding Information retrieval D-U1/U2 Accuracy Reasoning Data aggregation D-R1/R2/R3 Accuracy Arithmetic computation D-R4/R5 Accuracy Knowledge-Driven Understanding Clinical identification K-U1 AUC 1 1 footnotemark: 1 Reasoning Diagnostic assessment K-R1/R2 AUC Treatment planning K-R3 AUC Table 1: Overview of the 11 structured tasks in EHRStruct, categorized by scenario (Data-Driven vs. Knowledge-Driven) and cognitive level (Understanding vs. Reasoning). Data-Driven tasks include: D-U1/U2: data filtering based on field conditions; D-R1/R2/R3: value aggregation including count, average, and sum; D-R4/R5: arithmetic reasoning over numeric field trends. Knowledge-Driven tasks include: K-U1: clinical code identification; K-R1: mortality prediction; K-R2: disease prediction based on clinical profiles; K-R3: medication recommendation tailored to patient characteristics. To address these limitations, we introduce EHRStruct, a comprehensive benchmark specifically designed to systematically evaluate LLMs on structured EHR tasks. First, to expand task coverage, EHRStruct defines 11 diverse tasks across 6 categories, including information retrieval, data aggregation, and more. These categories are distilled from a thorough analysis of real-world clinical applications and prior research paradigms, ensuring that the selected tasks reflect both operational diversity and clinical relevance. Second, to mitigate dataset inconsistencies, we construct task-specific evaluation samples using standardized data sources from two complementary origins—the synthetic Synthea dataset ( 37 ) and the real-world eICU database ( 29 ) . For each task, we select representative and non-overlapping input–output pairs, validated through cross-checking by multiple domain experts, enabling reproducible and fair comparisons across models. Third, to resolve inconsistencies in input formatting and experimental protocols, we conduct a systematic exploration of input construction strategies and propose a unified framework that clearly distinguishes between different prompt structures for controlled experimentation. Finally, to enhance interpretability and support fine-grained diagnostic analysis, we further classify all tasks along two orthogonal dimensions: evaluation scenario—Data-Driven versus Knowledge-Driven, and cognitive complexity—understanding versus reasoning. This two-dimensional design enables deeper insights into the specific capabilities and limitations of LLMs. To assess the practicality and effectiveness of our proposed benchmark, EHRStruct, we evaluate 20 representative general and medical LLMs. We primarily focus on zero-shot settings, while also testing 1-shot, 3-shot, and 5-shot configurations to investigate few-shot performance across tasks in our benchmark. In addition, we systematically examine the impact of input formatting, comparing 4 common prompt structures for structured data, and evaluate the effectiveness of task-specific finetuning. In addition to evaluating base LLMs, we assess 11 representative LLM-based enhancement methods designed to improve structured data performance, including 8 originally developed for non-medical tasks and 3 specifically tailored for medical tasks. Building on insights from our evaluation of both base LLMs and enhancement methods, we propose EHRMaster—a novel code-augmented framework tailored for structured EHR tasks. EHRMaster operates in three stages: it first generates a high-level solution plan based on the task definition, then aligns key concepts in the plan with relevant data fields, and finally determines whether to generate executable code or proceed with direct reasoning. By structuring execution around task semantics, EHRMaster achieves substantial performance gains across diverse benchmark tasks. Our main contributions are as follows: • We introduce EHRStruct, a comprehensive benchmark featuring diverse clinically grounded tasks, standardized datasets, systematic input design, and interpretable evaluation to assess LLMs on structured EHR tasks. • We use EHRStruct to systematically evaluate 20 general and medical LLMs and 11 LLM-based enhancement methods, providing extensive analysis and insights into task-specific performance. • We propose EHRMaster, achieving state-of-the-art results on our benchmark and providing valuable inspiration for future research on structured EHR modeling. Figure 1: Overview of EHRStruct.