CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs

Raman Dutt, Pedro Sanchez, Yongcheng Yao, Steven McDonagh, Sotirios A. Tsaftaris, Timothy Hospedales · 2025

Read CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs by Raman Dutt, Pedro Sanchez, Yongcheng Yao, Steven McDonagh, Sotirios A. Tsaftaris, Timothy Hospedales (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10496.txt.

Introduction

Raman Dutt and co-authors build the first unified benchmark for AI-generated chest X-rays in a 2025 paper. Synthetic radiographs are a hot research area, but studies judge them with scattered protocols and rarely check privacy. Their benchmark grades generated images on three axes at once: realism, privacy risk, and usefulness for downstream tasks.

Over twenty metrics cover eleven leading text-to-image models, with plug-in support for newcomers. Even the best models stumbled on rare conditions, the tests showed. Privacy risk ran high regardless of image quality, and synthetic data helped classification but barely helped multimodal tasks. The authors publish baselines for all three axes and propose concrete directions. Code is public.

Catalog record

CREATORRaman Dutt, Pedro Sanchez, Yongcheng Yao, Steven McDonagh, Sotirios A. Tsaftaris, Timothy Hospedales
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10496

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs Raman Dutt, Pedro Sanchez, Yongcheng Yao, Steven McDonagh, Sotirios A. Tsaftaris, Timothy Hospedales Abstract We introduce CheXGenBench, a rigorous and multifaceted evaluation framework for synthetic chest radiograph generation that simultaneously assesses fidelity, privacy risks, and clinical utility across state-of-the-art text-to-image generative models. Despite rapid advancements in generative AI for real-world imagery, medical domain evaluations have been hindered by methodological inconsistencies, outdated architectural comparisons, and disconnected assessment criteria that rarely address the practical clinical value of synthetic samples. CheXGenBench overcomes these limitations through standardised data partitioning and a unified evaluation protocol comprising over 20 quantitative metrics that systematically analyse generation quality, potential privacy vulnerabilities, and downstream clinical applicability across 11 leading text-to-image architectures. Our results reveal critical inefficiencies in the existing evaluation protocols, particularly in assessing generative fidelity, leading to inconsistent and uninformative comparisons. Our framework establishes a standardised benchmark for the medical AI community, enabling objective and reproducible comparisons while facilitating seamless integration of both existing and future generative models. Additionally, we release a high-quality, synthetic dataset, SynthCheX-75K , comprising 75K radiographs generated by the top-performing model (Sana 0.6B) in our benchmark to support further research in this critical domain. Through CheXGenBench, we establish a new state-of-the-art and release our framework, models, and SynthCheX-75K dataset at https://raman1121.github.io/CheXGenBench/ 1 Introduction Recent advances in multi-modal generative models, particularly Text-to-Image (T2I) systems ramesh2021zero ; saharia2022photorealistic ; hurst2024gpt ; xie2025sana , have demonstrated remarkable capabilities in producing high-fidelity synthetic images that closely adhere to natural-language prompts. Central to this progress is the development of comprehensive, well-designed benchmarks that evaluate various aspects of generative performance. These benchmarks drive innovation by establishing standardised evaluation protocols and creating an equitable foundation for model comparison. The natural imaging domain has benefited from numerous such benchmarks, each meticulously assessing specific aspects and identifying limitations that researchers subsequently address in developing next-generation models. For example, MS-COCO dataset lin2014microsoft has been established as a seminal benchmark for evaluating general performance across multiple tasks, with particular emphasis on text-guided image generation. Building upon this foundation, more specialised benchmarks have emerged to assess specific attributes such as compositional understanding huang2023t2i ; huang2025t2i ; ghosh2023geneval , enabling more nuanced analysis of model capabilities. Despite the advancements in the natural imaging domain, there remains a significant gap in medical image analysis, and specifically in terms of benchmarking specialised tasks such as text-to-image generation. Figure 1 : Figure illustrating the overall schematic of the CheXGenBench benchmark for evaluating text-to-image models in synthetic radiograph generation. CheXGenBench organises evaluation into three dimensions: Fidelity (measuring generative quality and mode coverage), Privacy (assessing memorisation and re-identification risks), and Utility (evaluating practical value of synthetic samples through image classification and radiology report generation metrics) through 20+ metrics across 11 widely-adopted T2I models. Benchmarking Medical Imaging AI: Medical image analysis has made significant strides by benefiting from rapid advancements in artificial intelligence; however, its progress faces substantial constraints due to what has be characterised as a benchmarking crisis mahmood2025benchmarking . The ultimate goal of AI applications in medicine remains the development of intelligent systems capable of supporting and potentially transforming clinical decision-making processes. Progress toward this objective is frequently hindered by insufficient transparency in training and evaluation protocols, coupled with fragmented assessment criteria. Consequently, claims of “ state-of-the-art ” performance are often premature, non-reproducible, or reflect narrow contextual improvements rather than demonstrating genuine translational capability to clinical practice. The challenges of benchmarking in medical AI diagnostic tasks such as classification, localisation, segmentation and report generation have been discussed, and some improvements have been proposed irvin2019chexpert ; zong2023medfair ; karargyris2023medPerf ; zhang2024rexrank . However, despite opportunities for impactful societal applications from diagnostic training to rare condition simulation, benchmarking in medical image generation remains in an even more nascent and unsatisfactory state. This is exacerbated by unique challenges such as unclear evalution metrics, data scarcity due to privacy constraints, and long-tailed distributions of rare pathologies. While medical AI diagnostic tasks have been long studied, the generation of synthetic data through text prompts (medical descriptions) has recently emerged as a critical research focus. Besides being an interesting academic measure of medical AI capability, the ultimate motivation is that progress in diagnostic tasks is usually bottlenecked by data scarcity in the medical domain. High-fidelity generative models for synthetic data offer the promise to alleviate this bottleneck and ultimately facilitate clinical impact giuffre2023harnessing . Chest radiographs are the most commonly used frontline modality in medical imaging. Although the majority of collected clinical data remain inaccessible behind institutional firewalls and compliances mahmood2025benchmarking , several data repositories have been opened johnson2016mimic ; johnson2019mimic ; irvin2019chexpert ; zhang2025rexgradient , facilitating the development of generative models capable of synthesizing radiographs with potential relevance to downstream clinical utility ghalebikesabi2023differentially ; bluethgen2024vision ; lee2024llmcxr . Current research on Text-to-Image generation of radiographs can be broadly categorized into two primary streams: (1) studies prioritizing the enhancement of image fidelity and generative performance lee2023vision ; weber2023cascaded ; lee2024llmcxr ; dutt2024parameterefficient ; bluethgen2024vision ; han2024advancing ; huang2024chest ; moris2024adapted , and (2) investigations focusing on the mitigation of privacy concerns and patient re-identification risks fernandez2023privacy ; dar2023investigating ; akbar2025beware ; dutt2025memcontrol ; dutt2025devil that could undermine synthetic data utility. Despite constituting an active area of research with significant contributions, we have identified several critical benchmarking dimensions in which existing studies demonstrate consistent limitations. Minimal or Absent Comparative Baselines: Several notable studies either include minimal (self-proposed) dutt2024parameterefficient ; dutt2025memcontrol or no baselines weber2023cascaded for evaluation. han2024advancing ; lee2024llmcxr limit their comparative evaluations to only two competing methodologies. Reporting Inadequate Metrics: Studies throughout the existing literature predominantly report Fréchet Inception Distance (FID) heusel2017gans , specifically adopting in-domain image encoders based on primitive architectures Cohen2022xrv . Furthermore, no studies adequately report metrics characterising mode-coverage, a criterion of paramount importance in synthetic image generation for capturing the diversity of the underlying data distribution. Finally and most importantly, all metrics are reported as micro-averages across entire datasets without accounting for conditioning, let alone the long-tailed nature of medical data distributions. For example, excellent average generation fidelity could be reflective of the ability to generate more common images of healthy individuals, and the inability to generate images with pathologies of interest. This situation would offer limited clinical utility for pathology diagnosis. Restriction to Early-Stage Architectures: Existing studies bluethgen2024vision ; weber2023cascaded ; dutt2024parameterefficient ; favero2025conditional have predominantly confined themselves to outdated T2I model architectures Rombach_2022_CVPR , failing to address the crucial question of how recent advancements in generative modelling chen2024pixart ; xie2025sana ; flux2024 from the natural imaging domain translate to the specialised requirements of medical imaging contexts. Lack of a Unified Evaluation: Existing studies are fragmented between evaluating generative fidelity bluethgen2024vision ; weber2023cascaded ; lee2024llmcxr or privacy and re-identification risks fernandez2023privacy ; akbar2025beware ; dutt2025devil , failing to conduct and provide a unified evaluation of the two key aspects of synthetic radiograph generation research. Limited Evaluation on Synthetic Data Utility: Most studies fail to comment on the downstream utility of synthetic radiographs weber2023cascaded ; lee2024llmcxr often presenting generation results without rigorously assessing their potential impact on medical image analysis tasks such as classification, segmentation, or diagnostic reasoning. To address these critical limitations, we introduce CheXGenBench, a comprehensive benchmark designed

Read the complete text