MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-End Question Answering
Shaowei Guan, Yu Zhai, Hin Chi Kwok, Jiawei Du, Xinyu Feng, Jing Li, Harry Qin, Vivian Hui · 2026
Read MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-End Question Answering by Shaowei Guan, Yu Zhai, Hin Chi Kwok, Jiawei Du, Xinyu Feng, Jing Li, Harry Qin, Vivian Hui (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14265.txt.
Introduction
Guan and co-authors present MedPriv-Bench, a benchmark that scores privacy and usefulness together in medical question answering. Retrieval links to outside databases can leak patient identity through odd detail mixes, with no name ever shown. Human-guided agents build touchy medical contexts and matching questions. Across nine language models and three privacy methods, a clear trade-off appears. Fine-tuning roughly halves leakage while lifting answer quality, while local differential privacy cuts leakage most but drags quality down.
Catalog record
| CREATOR | Shaowei Guan, Yu Zhai, Hin Chi Kwok, Jiawei Du, Xinyu Feng, Jing Li, Harry Qin, Vivian Hui |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57228 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14265
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-End Question Answering Shaowei Guan, Yu Zhai, Hin Chi Kwok, Jiawei Du, Xinyu Feng, Jing Li, Harry Qin, Vivian Hui Abstract Recent advances in Retrieval-Augmented Generation (RAG) have enabled large language models (LLMs) to ground outputs in clinical evidence. However, connecting LLMs with external databases introduces the risk of contextual leakage : a subtle privacy threat where unique combinations of medical details enable patient re-identification even without explicit identifiers. Current benchmarks in healthcare heavily focus on accuracy, ignoring such privacy issues, despite strict regulations like Health Insurance Portability and Accountability Act (HIPAA) and General Data Protection Regulation (GDPR). To fill this gap, we present MedPriv-Bench , the first benchmark specifically designed to jointly evaluate privacy preservation and clinical utility in medical open-ended question answering. Our framework utilizes a multi-agent, human-in-the-loop pipeline to synthesize sensitive medical contexts and clinically relevant queries that create realistic privacy pressure. We establish a standardized evaluation protocol leveraging a pre-trained RoBERTa-Natural Language Inference (NLI) model as an automated judge to quantify data leakage, achieving an average of 85.9% alignment with human experts. Through an extensive evaluation of 9 representative LLMs, we demonstrate a pervasive privacy-utility trade-off. Our findings underscore the necessity of domain-specific benchmarks to validate the safety and efficacy of medical AI systems in privacy-sensitive environments. 1 Introduction The rapid advancement of large language models (LLMs) has expanded their use beyond general-purpose tasks into high-stakes domains such as healthcare, law, and education ( 7 ; 21 ; 9 ) . Despite their excellent generation and summarization capabilities, LLMs aroused a whirlpool of dubious concerns over “hallucinations”, i.e., generating factually incorrect or nonsensical ( 8 ) . In accuracy-critical settings like healthcare, where mistakes can directly impact patient outcomes, such hallucinations could lead to far-reaching consequences such as delayed treatment and early death. Retrieval-augmented generation (RAG) mitigates this issue by grounding model outputs in verifiable evidence retrieved from external knowledge sources ( 4 ) . However, integrating LLMs with external databases in healthcare introduces significant privacy challenges. Unlike the public corpora used for pre-training, healthcare RAG systems often retrieve from sensitive, locally hosted sources such as electronic health records (EHRs), clinical notes, and internal administrative documents. While such documents are necessary for accurate and context-aware assistance, retrieval and generation can create a risk of data leakage ( 5 ) . Although standard de-identification techniques, such as Named Entity Recognition (NER), effectively scrub explicit Protected Health Information (PHI) (e.g., patient names, addresses, medical record numbers) before indexing, they often fail to address the underlying semantic relationships within clinical narratives ( 20 ) . This work focuses on contextual leakage : a subtle privacy threat where an LLM leverages a patient’s rich clinical history to synthesize a response that, while devoid of explicit identifiers, contains a unique constellation of medical details (e.g., rare diagnosis combined with specific demographic or temporal markers) that enables re-identification ( 5 ; 18 ) . An illustrative example is shown in Figure 1 . Figure 1: Illustration of contextual leakage in healthcare retrieval-augmented generation (RAG) systems. If an LLM exposes sensitive information that enables re-identification, it violates patient rights and may fail to comply with regulations such as Health Insurance Portability and Accountability Act (HIPAA) and General Data Protection Regulation (GDPR) ( 2 ; 3 ) . Crucially, contextual leakage specifically breaches HIPAA’s “Minimum Necessary Standard”, which mandates that any PHI disclosure be limited to the minimum necessary for the intended purpose ( 2 ) . To isolate and measure this risk, we study a realistic deployment scenario where users ask benign, task-oriented clinical questions, evaluating whether resulting answers satisfy privacy requirements, even when explicit identifiers are removed. This differs from adversarial "red-teaming" approaches: rather than probing for worst-case vulnerabilities, we assess whether normal clinical use inadvertently enables re-identification through distinctive combinations of clinical details. Existing privacy benchmarks for LLMs predominantly focus on general-domain data leakage, leaving a critical void in dedicated evaluations for medical environments. While recent frameworks such as LLM-PBE ( 14 ) and PrivAuditor ( 29 ) systematically assess data privacy risks and general information leakage, they are primarily designed for broad, general-purpose applications. Other benchmarks, like PrivaCI-Bench ( 13 ) , evaluate contextual privacy and legal compliance, yet the field still lacks a standardized evaluation metric to quantify the complex dynamics of data leakage specifically within clinical contexts. Furthermore, as highlighted by prior research ( 5 ) , there exists an inherent trade-off between privacy and utility: rigorous privacy constraints often degrade clinical efficacy, while maximizing utility may inadvertently expose sensitive data. Consequently, a robust evaluation framework for medical LLMs should simultaneously account for both privacy preservation and clinical utility. Without a domain-specific benchmark that addresses these intersecting demands, researchers lack a standardized methodology to rigorously validate novel architectures, quantify their privacy-utility trade-offs, and establish equitable comparisons against existing models. To address these gaps, we propose the first benchmark designed to jointly assess privacy protection and clinical utility in medical open-ended question answering (QA). Additionally, we introduce a new evaluation approach using Natural Language Inference (NLI) and conduct an empirical study of prominent LLMs. Our contributions are as follows: 1. We introduce MedPriv-Bench , a novel multi-agent, human-in-the-loop pipeline for constructing a comprehensive benchmark designed to evaluate both privacy preservation and clinical utility in medical open-ended question answering. 2. We establish a standardized, scalable protocol for privacy evaluation by leveraging a pre-trained RoBERTa-NLI model as an automated judge, achieving 85.9% alignment with human expert judgment. 3. We conduct an extensive evaluation of 9 representative LLMs, including open-source, closed-source, and domain-specific medical LLMs using the MedPriv-Bench framework. 2 Related Work Medical QA Benchmarks. Benchmarks like PubMedQA ( 11 ) , MedQA ( 10 ) , and BioASQ ( 22 ) have set the standard for evaluating medical knowledge. However, they predominantly utilize discriminative formats (MCQ or Yes/No). As noted by Hager et al. ( 6 ) , discriminative metrics often overestimate a model’s clinical capability, as they allow models to guess correctly without proper reasoning. More importantly, these benchmarks are privacy-agnostic. A model could achieve 100% accuracy on MedQA while indiscriminately memorizing and regurgitating training data ( 12 ) , a behavior that would be catastrophic in a hospital setting. Our work shifts the focus from “What does the model know?” to “How safely can the model behave?” . Open-Ended Evaluation. Evaluating free-text generation in the medical domain presents significant challenges due to the inherent subjectivity of clinical reasoning and the variability of valid responses. Traditional n-gram based metrics (e.g., BLEU, ROUGE) correlate poorly with factual correctness in medical contexts, as they rely on surface-level lexical overlap and fail to capture semantic equivalence or clinical validity 28 ; 25 . These metrics often assign high scores to responses that are lexically similar to references yet factually incorrect or clinically misleading 25 ; 24 . Recent approaches have adopted LLM-as-a-Judge paradigms to better approximate human evaluation 27 . However, LLM judges in healthcare face limitations including bias, prompt sensitivity, and reduced reliability for subjective or nuanced clinical judgments 23 ; 19 . We advance this methodology by providing fine-grained scoring rubrics (Score Points) grounded in our synthetic injection methodology. This