CaseReportBench: An LLM Benchmark Dataset for Dense Information Extraction in Clinical Case Reports
Xiao Yu Cindy Zhang, Carlos R. Ferreira, Francis Rossignol, Raymond T. Ng, Wyeth Wasserman, Jian Zhu · 2025
Read CaseReportBench: An LLM Benchmark Dataset for Dense Information Extraction in Clinical Case Reports by Xiao Yu Cindy Zhang, Carlos R. Ferreira, Francis Rossignol, Raymond T. Ng, Wyeth Wasserman, Jian Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17265.txt.
Introduction
Xiao Yu Cindy Zhang and five co-authors release CaseReportBench, a benchmark for mining clinical case reports. Their 2025 paper targets dense extraction: sorting report facts into fixed categories. The focus is inborn errors of metabolism, rare diseases that are hard to diagnose. Case reports hold rich clues but sit unused by computers.
Experts annotated reports to build the test set. The team then graded several language models and prompting styles. Category-shaped prompts aligned best with the benchmark. An open model, Qwen2.5-7B, beat GPT-4o on this task. Clinicians judged the extracted details useful for diagnosis. Weak spots remain, notably missed negative findings that rule diseases out.
Catalog record
| CREATOR | Xiao Yu Cindy Zhang, Carlos R. Ferreira, Francis Rossignol, Raymond T. Ng, Wyeth Wasserman, Jian Zhu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17265
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CaseReportBench: An LLM Benchmark Dataset for Dense Information Extraction in Clinical Case Reports Xiao Yu Cindy Zhang, Carlos R. Ferreira, Francis Rossignol, Raymond T. Ng, Wyeth Wasserman, Jian Zhu Abstract Rare diseases, including Inborn Errors of Metabolism (IEM), pose significant diagnostic challenges. Case reports serve as key but computationally underutilized resources to inform diagnosis. Clinical dense information extraction refers to organizing medical information into structured predefined categories. Large Language Models (LLMs) may enable scalable information extraction from case reports but are rarely evaluated for this task. We introduce CaseReportBench , an expert-annotated dataset for dense information extraction of case reports (focusing on IEMs). Using this dataset, we assess various models and promptings, introducing novel strategies of category-specific prompting and subheading-filtered data integration . Zero-shot chain-of-thought offers little advantage over zero-shot prompting. Category-specific prompting improves alignment to benchmark. Open-source Qwen2.5:7B outperforms GPT-4o for this task. Our clinician evaluations show that LLMs can extract clinically relevant details from case reports, supporting rare disease diagnosis and management. We also highlight areas for improvement, such as LLMs’ limitations in recognizing negative findings for differential diagnosis. This work advances LLM-driven clinical NLP, paving the way for scalable medical AI applications. † † volume: 287 † † year: 2025 † † workshop: Conference on Health, Inference, and Learning (CHIL) 2025 Data and Code Availability This study used de-identified case reports from the non-commercial PMC Open Access Subset (CC BY-NC, CC BY-NC-SA, CC BY-NC-ND), accessed via PMC FTP 1 1 1 https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_bulk/oa_noncomm/xml/ on February 3, 2024. CaseReportBench is publicly available on Hugging Face 2 2 2 https://huggingface.co/datasets/cxyzhang/caseReportBench_ClinicalDenseExtraction_Benchmark , with source code on GitHub 3 3 3 https://github.com/cindyzhangxy/CaseReportBench , under a CC BY-NC license. Institutional Review Board (IRB) IRB approval was not required, as only publicly available, de-identified data were used. 1 Introduction Rare diseases encompass over 7,000 heterogeneous conditions that are often chronic, progressive, and debilitating ( Chung et al. 2022 ) . Among them, Inborn Errors of Metabolism (IEMs) is a group of over 1450 genetic disorders that disrupt metabolic pathways and contribute to significant morbidity and mortality ( Blau et al. 2014 ; Ferreira et al. 2021 ) . It affects approximately 1 in 2,500 births, posing a substantial global health burden ( Applegarth et al. 2000 ) . Case reports serve as key clinical documentation for rare diseases, offering actionable insights when robust guidelines are lacking. ( Carey 2010 ) . They are essential for researching novel disorders and uncovering unexpected disease-symptom associations ( Vandenbroucke 2001 ) . Extracting dense, granular information into a structured format can be fundamental for developing reliable AI-based diagnostic aids. However, manual extraction of such information is often error-prone, and impractical in busy clinical settings. Case reports typically follow a standard format, including title, abstract, introduction, case presentation, discussion, conclusion, and references ( Riley et al. 2017 ) . Specifically, the case presentation section provides a detailed account of patient assessment, such as system reviews and history, essential for disease diagnosis. This study aims to establish a benchmark for evaluating dense information extraction from clinical case reports, focusing on medical conditions, and identifying effective LLM-based methods for this task. Our contributions are as follows: • We present CaseReportBench , an expert-crafted medical benchmark designed for dense information extraction from case reports, and evaluate open-access LLMs and GPT-4o on this task. • We evaluate diverse prompting and data integration strategies for dense information extraction. We propose novel techniques, including category-specific extraction prompting and a subheading-filtered approach, to enhance dense information extraction by LLMs. • We conducted a clinical evaluation on LLM-extracted outputs, demonstrating their potential to reduce manual extraction efforts while identifying challenges. 2 Related Work Automating dense information extraction presents several challenges. Clinical text is often information-packed, complicating systematic extraction ( Landolsi et al. 2023 ; Wiest et al. 2024 ) . Extracting meaningful clinical insights requires linking multiple isolated entities to capture the full context ( Steinkamp et al. 2019 ) . Furthermore, an adaptable approach is required to accommodate the variety and complexity of information types—such as patient history, laboratory findings, and imaging ( Steinkamp et al. 2019 ) . Various benchmark datasets exist for specific medical NLP tasks. Datasets such as BIOSSES, containing sentence pairs from biomedical literature ( Soğancıoğlu et al. 2017 ) , and MedSTS, derived from EHRs ( Wang et al. 2020 ) , both serve as key benchmarks for medical sentence similarities. BioCreative V Chemical Disease Relation corpus, which annotates chemical-disease interactions from PubMed articles ( Li et al. 2016 ) , was used for named entity recognition. PMC-Patients dataset, which consists of patient summaries extracted from PubMed Central (PMC) ( Zhao et al. 2022 ) , provides a benchmark for a retrieval-based clinical decision support system. MedNLI, a dataset based on MIMIC-III, has been widely used for evaluating clinical natural language inference ( Romanov and Shivade 2018 ) . While existing benchmarks provide valuable insights, they primarily focus on isolated NLP tasks. In contrast, dense information extraction requires capturing structured knowledge across multiple dimensions. Relationships extracted in existing datasets (e.g., PMC-Patients ( Zhao et al. 2022 ) ) are often binary and insufficient for identifying multiple relations, attributes, and temporal aspects required by dense information extraction. Moreover, widely used benchmarks (e.g., MIMIC) ( Johnson et al. 2016 ) , focus on single-center EHR, limiting their applicability to diverse case report narratives. Thus, benchmarks specifically designed for dense information extraction from non-EHR clinical data remain largely unexplored. Our study seeks to overcome the limitations of existing benchmarks in dense information extraction. LLMs have proven effective in extracting fine-grained medical information from clinical narratives ( Fornasiere et al. 2024 ; Goel et al. 2023 ; Agrawal et al. 2022 ) . This study evaluates their capability for dense information extraction from medical case reports by testing various models, prompting strategies, and data integration methods. Additionally, we explore evaluation metrics for dense information extraction. 3 Dataset Construction Our goal is to extract clinically relevant insights from case reports, converting information-dense narratives into a structured format that aligns with real-world clinical assessments. To this end, We focused on 14 key clinical categories: Vitals and Hematology Findings ( Vitals_Hema ), Eyes, Ears, Nose, and Throat ( EENT ), Neurology ( NEURO ), Cardiovascular System ( CVS ), Respiratory System ( RESP ), Gastrointestinal System ( GI ), Genitourinary System ( GU ), Musculoskeletal System ( MSK ), Dermatology ( DERM ), Lymphatic System ( LYMPH ), Endocrinology ( ENDO ), Pregnancy , Laboratory and Imaging ( Lab_Image ), and History , as shown in Figure 1 , adapted from the standard Inpatient Work-Up and Monitoring Form 4 4 4 https://blogs.ubc.ca/oeetoolbox/2019/02/patient-work-up-from-sample-template-inpatient/ . Figure 1: Overview of CaseReportBench: Structuring Clinical Case Reports for Dense Information Extraction. This figure illustrates the systematic organization of case reports, the 14 key clinical categories used for structured extraction, and sample outputs from LLM-based extraction. 3.1 Dataset Preprocessing We adopted a privacy-conscious approach to derive the dataset from open-access case reports in PMC. Since case reports are often lengthy and contain subsections irrelevant to patient assessments, we developed a tailored preprocessing framework to filter case report sections. Specifically, case reports are divided into subsections based on their corresponding subheadings. We used regular expressions