Towards Selection of Large Multimodal Models as Engines for Burned-in Protected Health Information Detection in Medical Images
Tuan Truong, Guillermo Jimenez Perez, Pedro Osorio, Matthias Lenga · 2025
Read Towards Selection of Large Multimodal Models as Engines for Burned-in Protected Health Information Detection in Medical Images by Tuan Truong, Guillermo Jimenez Perez, Pedro Osorio, Matthias Lenga (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02014.txt.
Introduction
Tuan Truong, Guillermo Jimenez Perez, Pedro Osorio, and Matthias Lenga benchmark large multimodal models. The models act as detectors of protected health information burned into medical images. These are the names and identifiers printed directly onto scans. They must be scrubbed before sharing. Traditional pipelines chain optical character recognition with named entity recognition; the question is whether modern multimodal models do the whole job better.
Three models, GPT-4o, Gemini 2.5 Flash, and Qwen 2.5 7B, were tested in two configurations, text analysis alone and OCR plus semantic analysis. The multimodal models read text far more accurately than EasyOCR, with word error rates of only a few percent. Yet better reading did not reliably mean better privacy detection overall; the strongest gains appeared on cases with complex imprint patterns, while clean, high-contrast text made the pipelines equivalent. The authors close with empirically grounded selection advice for different operational constraints.
Catalog record
| CREATOR | Tuan Truong, Guillermo Jimenez Perez, Pedro Osorio, Matthias Lenga |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-02014
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Selection of Large Multimodal Models as Engines for Burned-in Protected Health Information Detection in Medical Images Tuan Truong, Guillermo Jimenez Perez, Pedro Osorio, Matthias Lenga Abstract The detection of Protected Health Information (PHI) in medical imaging is critical for safeguarding patient privacy and ensuring compliance with regulatory frameworks. Traditional detection methodologies predominantly utilize Optical Character Recognition (OCR) models in conjunction with named entity recognition. However, recent advancements in Large Multimodal Model (LMM) present new opportunities for enhanced text extraction and semantic analysis. In this study, we systematically benchmark three prominent closed and open-sourced LMMs, namely GPT-4o, Gemini 2.5 Flash, and Qwen 2.5 7B, utilizing two distinct pipeline configurations: one dedicated to text analysis alone and another integrating both OCR and semantic analysis. Our results indicate that LMM exhibits superior OCR efficacy (WER: 0.03-0.05, CER: 0.02-0.03) compared to conventional models like EasyOCR. However, this improvement in OCR performance does not consistently correlate with enhanced overall PHI detection accuracy. The strongest performance gains are observed on test cases with complex imprint patterns. In scenarios where text regions are well readable with sufficient contrast, and strong LMMs are employed for text analysis after OCR, different pipeline configurations yield similar results. Furthermore, we provide empirically grounded recommendations for LMM selection tailored to specific operational constraints and propose a deployment strategy that leverages scalable and modular infrastructure. † † address: Bayer AG, Berlin, Germany 1 Introduction In recent years, the detection of Protected Health Information (PHI) has become increasingly important, particularly within the realm of medical data management. Traditional pipelines for PHI detection typically consist of Optical Character Recognition (OCR) followed by analysis steps [ 1 , 2 , 3 , 4 , 5 ] . However, the advent of Large Multimodal Model (LMM) presents new opportunities for enhancing these processes. While LMM can be utilized in end-to-end pipelines for text analysis, their ability to localize text effectively for subsequent redaction remains a challenge [ 6 , 7 , 8 ] . This limitation stems from the fact that general LMMs are not specifically trained for multi-object localization, which increases the risk of incomplete text detection in medical images. In this paper, we build upon previous research of [ 6 , 7 ] by focusing on two specific configurations that leverage LMM effectively. The first configuration utilizes LMM exclusively during the text analysis phase, while the second extends their application to include text extraction. Research by [ 7 ] has demonstrated that these configurations offer competitive performance; however, integrating LMM into the OCR process significantly raises latency and costs, particularly when using advanced models like GPT-4o [ 9 ] . To further study these trade-offs, we introduce two additional models into our benchmarking: Gemini 2.5 Flash [ 10 ] and Qwen2.5-VL 7B [ 11 ] . Our main contribution is a systematic benchmark of three state-of-the-art LLMs under two PHI detection configurations, revealing key trade-offs between accuracy, latency, and deployability under varying resource constraints and privacy requirements. Specifically, our study addresses the following research questions: • What benefits, if any, can be achieved by replacing traditional OCR models with LMMs to enhance PHI detection? • Which LMM is optimal for selection with respect to performance metrics, latency considerations, and privacy implications? • What key factors should guide the deployment of an LMM-based PHI detection pipeline in real-world healthcare settings? 2 Methods In this study, we employ the PHI pipeline put forward by [ 7 ] , which comprises three integral components: text localization, text extraction, and text analysis (Figure ). The text localization module identifies text areas within images, while the text extraction module serves as an OCR engine, converting pixel-level text into machine-encoded text. The final component, text analysis, focuses on identifying PHI and related entities from the extracted text. To assess the performance of these models, we establish two distinct evaluation setups. In the first setup, each LMM operates exclusively as the text analysis module. In the second setup, the LMMs also function as the OCR engine, extracting text from localized image crops. This structured evaluation allows us to comprehensively analyze each model’s capabilities and their suitability for specific tasks within the PHI pipeline. To refine our candidate list of LMMs, we conducted a preliminary experiment to evaluate OCR capabilities of different LMMs and dedicated OCR models on medical image crops from RadPHI-test dataset (see Section 3.1 ). The RadPHI-test dataset was chosen due to its diverse imprint variations, which simulate real-world challenges in PHI detection. Apart from GPT-4o, which is a flagship model and was included in previous benchmarks [ 6 , 7 ] , the other models in the list were selected based on their suitability for specific operational needs. These include dedicated OCR models and lighter LMMs that prioritize low inference latency and cost-efficiency. The results, summarized in Table 1 , are characterized by word error rate (WER) and character error rate (CER). We subsequently shortlisted the top three models based on their performance across various aspects of our requirements. The first model, GPT-4o, emerges as a robust foundational model, achieving superior performance across a range of tasks. The second model, Gemini 2.5 Flash, is notable for its slower inference latency and lower operational costs while maintaining competitive performance compared to other leading proprietary models. This balance makes it a vital consideration in the selection of LMMs, especially for processes involving large batches of data. Lastly, the third model, Qwen2.5-VL 7B, is a lightweight, open-source option. Its ability to efficiently run a compact model is particularly advantageous in scenarios where data privacy is critical, such as in medical applications, as it is more cost-effective and easier to deploy than larger models. Model name WER \downarrow CER \downarrow GPT-4o [ 9 ] 0.03 0.02 Gemini-2.5-flash [ 10 ] 0.05 0.03 Qwen2.5-VL 7B [ 11 ] 0.05 0.03 InternVL2.5 8B [ 12 ] 0.09 0.10 TrOCR [ 13 ] 0.13 0.05 Qwen2.5-VL 3B [ 11 ] 0.13 0.06 GOT-OCR [ 14 ] 0.17 0.06 InternVL2.5 4B [ 12 ] 0.26 0.59 EasyOCR [ 15 ] 0.29 0.07 Table 1: Performance comparison of different models on the OCR task using the RadPHI-test dataset, ranked by WER. 3 Experiments 3.1 Data We use two datasets RadPHI-test and MIDI which were introduced in [ 7 ] and are publicly available 1 1 1 https://doi.org/10.5281/zenodo.17201610 . These datasets cover a range of imaging modalities, imprint variations, and text complexities to support robust benchmarking. RadPHI-test (Figure 1 a) includes 1000 radiological images equally distributed across four modalities with synthetic imprints generated at different locations, fonts, sizes, and contrasts to background. The MIDI dataset (Figure 1 b) is derived from the 2024 MIDI-B challenge [ 16 ] and includes images overlaid with DICOM header metadata using a standard DICOM viewer export. All images are labeled with bounding boxes and imprint categories. While MIDI presents challenges due to the presence of redundant PHI elements, like repeated dates and identifiers, the imprint variations are less pronounced compared to a the purely synthetic and heavily randomized imprints from RadPHI-test. The text categories that are considered PHI include patient names, addresses, identifiers, dates, phone numbers, and emails. Each image in RadPHI-test can have maximum eight imprints, whereas it is up to