Medical Image De-Identification Benchmark Challenge DOI: 10.59275/j.melba.2024-AAAA Volume: 2

\firstname Linmin \surname Pei \aff 1 \orcid 0000-0001-6135-9429, \name Granger Sutton \aff 2 \orcid 0000-0001-7498-8048, \name Michael Rutherford \aff 3 \orcid 0000-0003-2665-753X, \name Ulrike Wagner \aff 1 \orcid 0000-0002-3230-5058, \name Tracy Nolan \aff 3 \orcid 0000-0002-7023-7586, \name Kirk · 2025

Read Medical Image De-Identification Benchmark Challenge DOI: 10.59275/j.melba.2024-AAAA Volume: 2 by \firstname Linmin \surname Pei \aff 1 \orcid 0000-0001-6135-9429, \name Granger Sutton \aff 2 \orcid 0000-0001-7498-8048, \name Michael Rutherford \aff 3 \orcid 0000-0003-2665-753X, \name Ulrike Wagner \aff 1 \orcid 0000-0002-3230-5058, \name Tracy Nolan \aff 3 \orcid 0000-0002-7023-7586, \name Kirk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23608.txt.

Introduction

A large multi-team group ran a public contest to strip private details from medical images, reported in this 2025 paper. Scans shared for research can carry burned-in text: names, dates, and scanner notes baked into the pixels. Removing that text by hand is slow, and missing a piece risks a patient's privacy.

The benchmark challenge gave teams the same sets of images and scored how well their tools found and erased the hidden text. Entries came from labs across several countries, each with its own approach, from classic image processing to deep learning. The report gathers the field's methods and results in one place. A shared yardstick matters here: hospitals and archives need proof, not promises, before they trust a tool with real patient scans.

Catalog record

CREATOR\firstname Linmin \surname Pei \aff 1 \orcid 0000-0001-6135-9429, \name Granger Sutton \aff 2 \orcid 0000-0001-7498-8048, \name Michael Rutherford \aff 3 \orcid 0000-0003-2665-753X, \name Ulrike Wagner \aff 1 \orcid 0000-0002-3230-5058, \name Tracy Nolan \aff 3 \orcid 0000-0002-7023-7586, \name Kirk
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23608

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Medical Image De-Identification Benchmark Challenge DOI: 10.59275/j.melba.2024-AAAA Volume: 2 \firstname Linmin \surname Pei \aff 1 \orcid 0000-0001-6135-9429, \name Granger Sutton \aff 2 \orcid 0000-0001-7498-8048, \name Michael Rutherford \aff 3 \orcid 0000-0003-2665-753X, \name Ulrike Wagner \aff 1 \orcid 0000-0002-3230-5058, \name Tracy Nolan \aff 3 \orcid 0000-0002-7023-7586, \name Kirk Smith \aff 3 \orcid 0000-0002-8735-7576, \name Phillip Farmer \aff 3 \orcid 0000-0003-1448-1346, \name Peter Gu \aff 5, \name Ambar Rana \aff 5, \name Kailing Chen \aff 5, \name Thomas Ferleman \aff 5, \name Brian Park \aff 5, \name Ye Wu \aff 5, \name Jordan Kojouharov \aff 6, \name Gargi Singh \aff 6, \name Jon Lemon \aff 6, \name Tyler Willis \aff 6, \name Milos Vukadinovic \aff 7, 8 \orcid 0000-0002-0431-1493, \name Grant Duffy \aff 8, \name Bryan He \aff 9 \orcid 0000-0002-6150-761X, \name David Ouyang \aff 8 \orcid 0000-0002-3813-7518, \name Marco Pereañez \aff 10 \orcid 0000-0002-6447-4956, \name Daniel Samber \aff 10 \orcid 0000-0002-9241-2330, \name Derek A. Smith \aff 10 \orcid 0000-0002-8879-281X, \name Christopher Cannistraci \aff 10 \orcid 0009-0003-0900-8252, \name Zahi Fayad \aff 10 \orcid 0000-0002-3439-7347, \name David S. Mendelson \aff 11 \orcid 0000-0002-1555-8002, \name Michele Bufano \aff 12 \orcid 0009-0000-5067-9814, \name Elmar Kotter \aff 12 \orcid 0000-0001-9022-6000, \name Hamideh Haghiri \aff 13 \orcid 0009-0006-7308-4235, \name Rajesh Baidya \aff 13 \orcid 0009-0003-5235-1769, \name Stefan Dvoretskii \aff 13 \orcid 0000-0001-7769-0167, \name Klaus H. Maier-Hein \aff 13, 14 \orcid 0000-0002-6626-2463, \name Marco Nolden \aff 13, 14 \orcid 0000-0001-9629-0564, \name Christopher Ablett \aff 15, \name Silvia Siggillino \aff 16, \name Sandeep Kaushik \aff 17 \orcid 0000-0003-0654-0799, \name Hongzhu Jiang \aff 18, \name Sihan Xie \aff 18, \name Zhiyu Wan \aff 18, 19 \orcid 0000-0003-3752-5778, \name Alex Michie \aff 20 \orcid 0000-0003-1685-6092, \name Simon J Doran \aff 20 \orcid 0000-0001-8569-9188, \name Angeline Aurelia Waly \aff 21, \name Felix A. Nathaniel Liang \aff 21, \name Humam Arshad Mustagfirin \aff 21, \name Michelle Grace Felicia \aff 21, \name Kuo Po Chih \aff 21, \name Rahul Krish \aff 22, \name Ghulam Rasool \aff 23 \orcid 0000-0001-8551-0090, \name Nidhal Bouaynaya \aff 25, \name Nikolas Koutsoubis \aff 23, 24, \name Kyle Naddeo \aff 25, \name Kartik Pandit \aff 24, \name Tony O’Sullivan \aff 22, \name Raj Krish \aff 22, \name Qinyan Pan \aff 26 \orcid 0009-0006-8294-6167, \name Scott Gustafson \aff 26, \name Benjamin Kopchick \aff 27 \orcid 0000-0003-1125-0155, \name Laura Opsahl-Ong \aff 27, \name Andrea Olvera-Morales \aff 27, \name Jonathan Pinney \aff 27, \name Kathryn Johnson \aff 27, \name Theresa Do \aff 27, \name Juergen Klenk \aff 27, \name Maria Diaz \aff 28, \name Arti Singh \aff 28, \name Rong Chai \aff 28 \orcid 0000-0003-3049-7241, \name David A. Clunie \aff 4 \orcid 0000-0002-2406-1145, \name Fred Prior \aff 3 \orcid 0000-0002-6314-5683, \name Keyvan Farahani \aff * \orcid 0000-0003-2111-1896 Abstract The de-identification (deID) of protected health information (PHI) and personally identifiable information (PII) is a fundamental requirement for sharing medical images, particularly through public repositories, to ensure compliance with patient privacy laws. In addition, preservation of non-PHI metadata to inform and enable downstream development of imaging artificial intelligence (AI) is an important consideration in biomedical research. In 2024, the National Cancer Institute (NCI), in collaboration with the Society for Medical Image Computing and Computer Assisted Intervention (MICCAI), and Sage Bionetworks, conducted the Medical Image de-identification Benchmark (MIDI-B) Challenge. The goal of MIDI-B was to provide a standardized platform for benchmarking of DICOM image deID tools based on a set of rules conformant to the HIPAA Safe Harbor regulation, the DICOM Attribute Confidentiality Profiles, and best practices in preservation of research-critical metadata, as defined by The Cancer Imaging Archive (TCIA). The challenge employed a large, diverse, multi-center, and multi-modality set of real de-identified radiology images with synthetic PHI/PII inserted. The MIDI-B Challenge consisted of three phases: training, validation, and test. Eighty individuals registered for the challenge. In the training phase, we encouraged participants to tune their algorithms using their in-house or public data. The validation and test phases utilized the DICOM images containing synthetic identifiers (of 216 and 322 subjects, respectively). Ten teams successfully completed the test phase of the challenge. To measure success of a rule-based approach to image deID, scores were computed as the percentage of correct actions from the total number of required actions. The scores ranged from 97.91% to 99.93%. Participants employed a variety of open-source and proprietary tools with customized configurations, large language models, and optical character recognition (OCR). In this paper we provide a comprehensive report on the MIDI-B Challenge’s design, implementation, results, and lessons learned. \melbaid YYYY:NNN \melbaauthors Keyvan Farahani and Linmin Pei \firstpageno 1 \melbayear 2025 \datesubmitted yyyy-m1-d1 \datepublished yyyy-m2-d2 \melbaspecialissue Medical Imaging with Deep Learning (MIDL) 2020 \melbaspecialissueeditors Marleen de Bruijne, Tal Arbel, Ismail Ben Ayed, Hervé Lombaert \ShortHeadings MELBA Journal Sample ArticleKeyvan Farahani and Linmin Pei \affiliations \num 1 \addr Frederick National Laboratory for Cancer Research, Frederick, MD, 21702, USA \num 2 \addr National Cancer Institute, National Institute of Health (NIH), Bethesda, MD 20892, USA \num 3 \addr University of Arkansas for Medical Sciences, Little Rock, Arkansas, USA \num 4 \addr PixelMed Publishing, Bangor, PA, USA \num 5 \addr Essential Software Inc. 9711 Washingtonian Blvd, Suite 550, Gaithersburg, MD, 20878 \num 6 \addr Amazon Web Services (AWS) \num 7 \addr University of California, Los Angeles, CA, USA \num 8 \addr Cedars-Sinai Medical Center, Los Angeles, CA, USA \num 9 \addr Stanford University, Palo Alto, CA, USA \num 10 \addr Biomedical Engineering & Imaging Institute, Icahn School of Medicine at Mount Sinai, New York NY 10029, USA \num 11 \addr Department of Radiology, Icahn School of Medicine at Mount Sinai, New York NY 10029, USA \num 12 \addr Center for Diagnostic and Therapeutic Radiology, Department of Radiology, University Medical Center Freiburg, Hugstetter Straße 55, D-79106 Freiburg, Germany \num 13 \addr Division of Medical and Biological Informatics, German Cancer Research Center, Heidelberg 69120, Germany \num 14 \addr Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg 69120, Germany \num 15 \addr GE Healthcare, Dublin, Ireland \num 16 \addr GE Healthcare, Milano, Italy \num 17 \addr GE Healthcare, Waukesha, USA \num 18 \addr ShanghaiTech University, Shanghai 201210, China \num 19 \addr Vanderbilt University Medical Center, Nashville TN 37203, USA \num 20 \addr Division of Radiotherapy and Imaging, Institute of Cancer Research, London UK \num 21 \addr National Tsing Hua University, Hsinchu 300044, Taiwan \num 22 \addr Impact Business Information Solutions, Inc, Princeton NJ 08540, USA \num 23 \addr Moffitt Cancer Center, Tampa FL 33612, USA \num 24 \addr University of South Florida, Tampa FL 33620, USA \num 25 \addr Rowan University, Glassboro NJ 08028, US \num 26 \addr Ellumen, Inc. Silver Spring, MD 20910 \num 27 \addr Deloitte Consulting LLP, New York, NY, USA \num 28 \addr Sage Bionetworks \num * \addr (Corresponding Author) National Heart, Lung, and Blood Institute, National Institute of Health (NIH), Bethesda, MD 20892, USA keywords Medical image de-identification, protected health information, personally identifiable information, HIPAA Safe Harbor regulation, multi-center, multi-modality, synthetic PHI, DICOM de-identification, optical character recognition (OCR). 1 Introduction \enluminure A fundamental requirement for sharing medical images, particularly through public image repositories that are openly accessible without restrictions, is the de-identification (deID) of protected health information (PHI) and personally identifiable information (PII) from the Digital Imaging and Communications in Medicine (DICOM) header and the pixel matrix of all images.

Read the complete text