CUPCase: Clinically Uncommon Patient Cases and Diagnoses Dataset
Oriel Perets, Ofir Ben Shoham, Nir Grinberg, Nadav Rappoport · 2025
Read CUPCase: Clinically Uncommon Patient Cases and Diagnoses Dataset by Oriel Perets, Ofir Ben Shoham, Nir Grinberg, Nadav Rappoport (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06204.txt.
Introduction
Oriel Perets, Ofir Ben Shoham, Nir Grinberg, and Nadav Rappoport published this medical data paper in 2025. It introduces CUPCase, a benchmark set built from 3,562 real case reports published by BMC journals. The cases are the awkward ones doctors actually face: rare diseases, common diseases showing up in odd ways, and treatments that surprise. Each case comes with its diagnosis in free text plus a multiple-choice version with wrong options mixed in.
The team used the set to test how well leading language models can name the right diagnosis. A general-purpose model, GPT-4o, beat the medical models on both tasks, reaching 87.9 percent on the choice task. It also kept most of its score when shown only the first fifth of a case. The authors offer the set as an open way to judge models meant to support clinical decisions.
Catalog record
| CREATOR | Oriel Perets, Ofir Ben Shoham, Nir Grinberg, Nadav Rappoport |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06204
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CUPCase: Clinically Uncommon Patient Cases and Diagnoses Dataset Oriel Perets, Ofir Ben Shoham, Nir Grinberg, Nadav Rappoport Abstract Medical benchmark datasets significantly contribute to developing Large Language Models (LLMs) for medical knowledge extraction, diagnosis, summarization, and other uses. Yet, current benchmarks are mainly derived from exam questions given to medical students or cases described in the medical literature, lacking the complexity of real-world patient cases that deviate from classic textbook abstractions. These include rare diseases, uncommon presentations of common diseases, and unexpected treatment responses. Here, we construct C linically U ncommon P atient C ases and Diagnosis Dataset (CUPCase) based on 3,562 real-world case reports from BMC, including diagnoses in open-ended textual format and as multiple-choice options with distractors. Using this dataset, we evaluate the ability of state-of-the-art LLMs, including both general-purpose and Clinical LLMs, to identify and correctly diagnose a patient case, and test models’ performance when only partial information about cases is available. Our findings show that general-purpose GPT-4o attains the best performance in both the multiple-choice task (average accuracy of 87.9%) and the open-ended task (BERTScore F1 of 0.764), outperforming several LLMs with a focus on the medical domain such as Meditron-70B and MedLM-Large. Moreover, GPT-4o was able to maintain 87% and 88% of its performance with only the first 20% of tokens of the case presentation in multiple-choice and free text, respectively, highlighting the potential of LLMs to aid in early diagnosis in real-world cases. CUPCase expands our ability to evaluate LLMs for clinical decision support in an open and reproducible manner. Introduction Large Language Models (LLMs) have demonstrated promising results in the medical field ( Zhou et al. 2023 ) , including when general-purpose LLMs (such as GPT-4) are applied to medical tasks. For example, GPT-4 showed promising performance in the United States Medical Licensing Examination ( Nori et al. 2023 ) , interpreting medical concepts (diagnoses, procedures, and drug codes) ( Shoham and Rappoport 2024b ) , rare disease prediction ( do Olmo et al. 2024 ) , and more. Clinical LLMs (CLLMs) are specialized LLMs focused on the clinical domain, as opposed to general-purpose LLMs. Examples of CLLMs include Meditron ( Chen et al. 2023 ) , MedLM ( Singhal et al. 2023 ) , BioMistral ( Labrak et al. 2024 ) , and Llama3-OpenBioLLM ( Ankit Pal 2024 ) . CLLMs have several applications, such as disease prediction ( Shoham and Rappoport 2024a ) , medical chatbots, healthcare education, text generation, and more ( He et al. 2024 ) . To evaluate the performance of LLMs and CLLMs, medical clinical benchmarks are required. Notable examples of existing benchmarks in the medical domain include PubMedQA ( Jin et al. 2019 ) , which focuses on question-answering over PubMed-derived questions; ClinicalBench ( Liu et al. 2024 ) , which covers tasks such as treatment recommendation and hospitalization summarization; and BioASQ-QA, which involves biomedical question-answering, document retrieval, text snippet extraction, and summarization ( Krithara et al. 2023 ) . Additionally, some benchmarks are specifically designed for medical exam questions, such as MedQA ( Jin et al. 2021 ) and MedMCQA, which are multiple-choice QA datasets tailored to medical entrance exam scenarios ( Pal, Umapathi, and Sankarasubbu 2022 ) . Given their origin, these medical benchmarks often focus on evaluating LLMs’ knowledge based on standard medical literature and fail to assess many skills necessary for deployment in a realistic clinical decision-making environment ( Hager et al. 2024 ; Mehandru et al. 2024 ) . In particular, showed that only 5% of reviewed works used real patient care data for LLM evaluation ( Bedi et al. 2024 ) . To complement these previous efforts, we propose using published case reports as a benchmark for LLMs. Clinical case reports are detailed accounts of individual patient cases that highlight unique or rare conditions, treatments, or outcomes, often providing insights into new or unusual aspects of medical practice. Published in scientific journals, they serve as valuable educational resources, contributing to medical knowledge by documenting specific clinical scenarios that may not be covered in larger studies. Published patient case reports offer crucial insights beyond the scope of traditional textbooks. These reports can reveal significant scientific observations that might be missed in clinical trials, and expand our understanding by introducing novel findings and scenarios that deviate from classical cases ( Cohen 2006 ) . Figure 1: Schematic flow of the dataset curation process. Related Work Several benchmarks were designed specifically to evaluate the performance of LLMs in rare diseases. For instance, ( Reese et al. 2024 ) assessed the diagnostic accuracy of GPT-4 on 5,000 rare disease cases. RareBench ( Chen et al. 2024 ) is a recent benchmark developed for rare diseases, demonstrating the potential of integrating LLMs into the clinical diagnostic process for these conditions. Another benchmark, RareDis ( Martínez-deMiguel et al. 2022 ) , includes 1,041 patient cases. Additionally, ( do Olmo et al. 2024 ) found that GPT-4 and Claude achieved the highest performance in diagnosing rare diseases, based on evaluations of 200 synthetic patient cases and 275 publicly available patient cases. However, our proposed benchmark is not specific only to rare diseases. Although existing previously published clinical benchmarks of rare diseases identification, these benchmarks do not address other complex patient cases, such cases may include uncommon presentations of common diseases or unexpected treatment responses. ( Ríos-Hoyo et al. 2024 ) examined complex medical cases from a single center (Massachusetts General Hospital Case Records) and asked GPT-4 to generate diagnoses. They found that GPT-4 provided the correct diagnosis on the first attempt in 42% of the cases. However, they did not evaluate CLLMs, and their dataset consisted of only 75 medical cases. Another relevant work by ( Kanjee, Crowe, and Rodman 2023 ) explored complex diagnostic identification beyond rare diseases. They applied GPT-4 to 70 complex cases from BMJ New England Journal of Medicine clinicopathologic conferences. Their dataset was extracted from these conferences and included cases with final pathological diagnoses used for educational purposes. They found that GPT-4 achieved an accuracy of 39% on the first prediction attempt in their evaluation of open questions. In contrast to previous studies, our work introduces a substantial dataset for evaluation, comprising 3,562 patient case presentations across a diverse array of clinical conditions, significantly expanding beyond the sample of about 75 cases in prior work. Moreover, our dataset is designed for scalability, allowing for easy expansion with additional case presentations from various journals using our generic open-source code. Importantly, our dataset encompasses a wide spectrum of scenarios, including rare diseases, uncommon presentations, and complex cases that deviate from classic medical knowledge. The dataset is also not confined to a single medical center, geographic location, or specific clinical practice (i.e. Rheumatology, Oncology, etc.). Along with the newly constructed benchmark dataset, we conducted a thorough evaluation of current state-of-the-art LLMs, both specialized for the medical domain and general-purpose and included open and closed-source models. We evaluated the models in two tasks using zero-shot learning. The first task involved multiple-choice questions, where strong distractors were introduced by selecting choices with high semantic similarity. The second evaluation task required the models to generate a response to an open-ended diagnosis question, which emulates more closely the unconstrained