MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes
Asma Ben Abacha *, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, Thomas Lin · 2024
Read MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes by Asma Ben Abacha *, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, Thomas Lin (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19260.txt.
Introduction
Asma Ben Abacha and six co-authors release MEDEC, a benchmark for catching errors in clinical notes. Their 2024 paper targets a gap in testing medical language models. Models ace exam questions, yet no one checked them as proofreaders of real notes. MEDEC fills that gap with thousands of texts.
The set holds three thousand eight hundred forty-eight clinical texts. Five error kinds are covered, from wrong diagnoses to wrong drugs. Fresh notes from three hospital systems were kept unseen by any model. Seventeen teams competed on the set in a shared task. Top commercial models were then graded, and two doctors took the same test. The benchmark proved hard enough to separate strong from weak checkers.
Catalog record
| CREATOR | Asma Ben Abacha *, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, Thomas Lin |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-19260
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes Asma Ben Abacha *, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, Thomas Lin Abstract Several studies showed that Large Language Models (LLMs) can answer medical questions correctly, even outperforming the average human score in some medical exams. However, to our knowledge, no study has been conducted to assess the ability of language models to validate existing or generated medical text for correctness and consistency. In this paper, we introduce MEDEC 1 1 1 https://github.com/abachaa/MEDEC , the first publicly available benchmark for medical error detection and correction in clinical notes, covering five types of errors (Diagnosis, Management, Treatment, Pharmacotherapy, and Causal Organism). MEDEC consists of 3,848 clinical texts, including 488 clinical notes from three US hospital systems that were not previously seen by any LLM. The dataset has been used for the MEDIQA-CORR shared task to evaluate seventeen participating systems ( Ben Abacha et al., 2024 ) . In this paper, we describe the data creation methods and we evaluate recent LLMs (e.g., o1-preview, GPT-4, Claude 3.5 Sonnet, and Gemini 2.0 Flash) for the tasks of detecting and correcting medical errors requiring both medical knowledge and reasoning capabilities. We also conducted a comparative study where two medical doctors performed the same task on the MEDEC test set. The results showed that MEDEC is a sufficiently challenging benchmark to assess the ability of models to validate existing or generated notes and to correct medical errors. We also found that although recent LLMs have a good performance in error detection and correction, they are still outperformed by medical doctors in these tasks. We discuss the potential factors behind this gap, the insights from our experiments, the limitations of current evaluation metrics, and share potential pointers for future research. August 24, 2026 1 Introduction A survey study from US health care organizations showed that 1 in 5 patients who read a clinical note reported finding a mistake and 40% perceived the mistake as serious, with the most common category of mistakes being related to current or past diagnoses ( Bell et al., 2020 ) . Figure 1: Examples from the MEDEC dataset. On the other hand, more and more medical documentation tasks (e.g., clinical note generation) are being supported by LLMs. In multiple studies, LLMs have shown the ability to answer accurately questions from medical exams ( Gilson et al., 2023 ; Johnson et al., 2023 ; Schubert et al., 2023 ) and to imitate clinical reasoning in providing diagnoses ( Savage et al., 2024 ) . However, one of the main obstacles in adopting LLMs in medical documentation tasks is their potential to generate hallucinations or incorrect information ( Tang et al., 2023 ) and harmful content that might alter clinical decision making ( Chen et al., 2024 ) . Rigorous validation methods are essential to mitigate these risks and make LLMs safer to use for medical content generation ( Karabacak and Margetis, 2023 ) . Such validation requires relevant benchmarks to assess whether it can be fully automated with validation models. A key task in this validation is the ability to detect and correct medical errors in clinical texts. Most previous studies on (common sense) error detection have focused on the general domain ( Wang et al., 2020 ; Onoe et al., 2021 ) . In this paper, we tackle the problem of identifying and correcting medical errors in clinical texts. From a human perspective, identifying and correcting these errors requires medical expertise, specialized knowledge, and sometimes practical experience. We introduce a new dataset, MEDEC, and experiment with different recent LLMs (e.g., Claude 3.5 Sonnet, o1-preview, and Gemini 2.0 Flash). To the best of our knowledge, this is the first publicly available benchmark and study on automatic error detection and correction in clinical notes. 2 Related Work Jang et al. (2022) introduced a benchmark for consistency evaluation and evaluated pretrained language models (e.g, BERT, T5, and GPT-2) on three main categories: semantic, logical, and factual consistency. They found that those language models do not perform well in every test case and have a high level of inconsistency in many cases. Jang and Lukasiewicz (2023) investigated the trustworthiness of more recent language models, ChatGPT and GPT-4, regarding semantic consistency and found that while both models appear to show an enhanced language understanding and reasoning ability, they often fail at generating logically consistent predictions. In the medical domain, several recent studies evaluated large language models’ accuracy and consistency. Johnson et al. (2023) conducted a study to assess the accuracy and reliability of medical responses generated by chatGPT. Thirty-three physicians across 17 specialties generated 284 medical questions with different levels of difficulty and graded ChatGPT’s answers for accuracy and completeness. While most of the generated text was evaluated by physicians as accurate, there were potential limitations in handling complex medical questions. In two separate studies, Schubert et al. (2023) and Gilson et al. (2023) found that GPT models can answer medical questions correctly in neurology board–style examinations and the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, even outperforming the average human score in some instances. Chen et al. (2024) assessed the effect and safety of LLM-assisted patient messaging, as one of the earliest applications of LLMs in electronic health records (EHRs). The increased similarity of response content between LLM drafts and LLM-assisted responses showed that doctors might adopt the LLM’s assessment, and that in a minority of LLM drafts, if left unedited, could lead to severe harm or death. The safe introduction and use of LLMs in medical documentation tasks requires reliable and automatic validation methods. However, as far as we know, no benchmark was made publicly available to assess the ability of LLMs in validating existing or generated medical text for correctness and consistency. In this paper, we present MEDEC, the first benchmark for medical error detection and correction in clinical notes. We describe the data creation methods and we evaluate recent state-of-the-art open domain LLMs for these tasks. The MEDEC dataset has been used in the first shared task on medical error detection and correction, MEDIQA-CORR 2024, to evaluate models and solutions from seventeen participating teams ( Ben Abacha et al., 2024 ) . 3 MEDEC Dataset We created a new dataset of 3,848 clinical texts from different specialties. Eight medical annotators participated in the annotation task. The dataset covers five types of errors, selected after analyzing the most frequent types of questions asked in medical board exams. The error types are: • Diagnosis - The provided diagnosis is inaccurate. • Management - The next step provided in management is inaccurate. • Pharmacotherapy - The recommended pharmacotherapy is inaccurate. • Treatment - The recommended treatment is inaccurate. • CausalOrganism - The indicated causal organism or causal pathogen is inaccurate. Figure 1 presents examples from the MEDEC dataset. Each clinical text is either correct or contains one error created using one of two different methods: M#1 (MS) & M#2 (UW). 3.1 Data Creation Method #1 (MS) In this method, we leverage medical board exams from the MedQA collection ( Jin et al., 2020 ) . Four annotators with medical backgrounds used the medical narratives