Perceptual Evaluation of GANs and Diffusion Models for Generating X-rays

Gregory Schuit, Denis Parra, Cecilia Besa · 2025

Read Perceptual Evaluation of GANs and Diffusion Models for Generating X-rays by Gregory Schuit, Denis Parra, Cecilia Besa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07128.txt.

Introduction

Gregory Schuit, Denis Parra, and Cecilia Besa judge fake chest X-rays by eye in two thousand twenty-five. Image generators could ease data scarcity in medical imaging. Rare ills are hard to learn from without more cases. But poor fakes can harm trust and model skill.

The team made X-rays for four chest ills with two model types. Real images came from a known hospital data set. Three radiologists tried to spot fakes and check if traits fit the named ill. Diffusion models made more real looking images on the whole. The rival type won on some traits. Cues the doctors used expose gaps in both.

Catalog record

CREATORGregory Schuit, Denis Parra, Cecilia Besa
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07128

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Perceptual Evaluation of GANs and Diffusion Models for Generating X-rays Gregory Schuit, Denis Parra, Cecilia Besa Abstract Generative image models have achieved remarkable progress in both natural and medical imaging. In the medical context, these techniques offer a potential solution to data scarcity—especially for low-prevalence anomalies that impair the performance of AI‑driven diagnostic and segmentation tools. However, questions remain regarding the fidelity and clinical utility of synthetic images, since poor generation quality can undermine model generalizability and trust. In this study, we evaluate the effectiveness of state-of-the-art generative models—Generative Adversarial Networks (GANs) and Diffusion Models (DMs)—for synthesizing chest X-rays conditioned on four abnormalities: Atelectasis (AT), Lung Opacity (LO), Pleural Effusion (PE), and Enlarged Cardiac Silhouette (ECS). Using a benchmark composed of real images from the MIMIC-CXR dataset and synthetic images from both GANs and DMs, we conducted a reader study with three radiologists of varied experience. Participants were asked to distinguish real from synthetic images and assess the consistency between visual features and the target abnormality. Our results show that while DMs generate more visually realistic images overall, GANs can report better accuracy for specific conditions, such as absence of ECS. We further identify visual cues radiologists use to detect synthetic images, offering insights into the perceptual gaps in current models. These findings underscore the complementary strengths of GANs and DMs and point to the need for further refinement to ensure generative models can reliably augment training datasets for AI diagnostic systems. 1 Introduction Recent advances in generative AI have led to remarkable progress in image synthesis across domains, including medical imaging. These models—particularly Generative Adversarial Networks (GANs) [ 3 ] and Diffusion Models (DMs) [ 7 ] —have the potential to address longstanding challenges in medical AI, such as data scarcity for low-prevalence conditions and the generation of counterfactual explanations for model interpretability. While Diffusion Models have recently surpassed GANs in image quality and are gaining popularity in chest X-ray (CXR) generation [ 6 ] , their clinical reliability remains underexplored. Most existing studies evaluate these models using proxy metrics like Frechet Inception Distance (FID) [ 8 ] , which do not capture radiological fidelity or diagnostic relevance. Although some works include expert feedback, they often rely on small image sets or overlook the reasons behind radiologists’ judgments. To address this gap, we conduct a human-centered evaluation comparing GAN- and DM-generated chest X-rays. We focus on four common abnormalities: Atelectasis, Lung Opacity, Pleural Effusion, and Enlarged Cardiac Silhouette. Our study involves two structured user studies with three board-certified radiologists, who assess image realism and the presence of the intended pathology. We also collect qualitative insights on the visual cues used to identify synthetic content. Our contributions are twofold: (1) we extend prior evaluation frameworks with a detailed, anomaly-specific analysis based on expert assessments, and (2) we offer a direct comparison between two state-of-the-art generative models, revealing trade-offs in realism versus conditional accuracy. Our findings highlight the complementary strengths of GANs and DMs and underscore the importance of human-in-the-loop validation in medical image synthesis. To foster reproducibility, we release the source code 1 1 1 https://github.com/gregschuit/radiologist-perceptual-eval-xrays , labeling interface, and generated dataset. This work aims to guide future development of trustworthy generative models for medical imaging applications. 2 Related Work Generative models for medical imaging—especially GANs [ 3 ] and Diffusion Models (DMs) [ 7 ] —have shown promising results. GANs train through adversarial feedback between a generator and a discriminator, while DMs iteratively denoise random noise into images, achieving high fidelity but often at greater computational cost [ 2 ] . In synthetic medical image evaluation, Segal et al. [ 8 ] introduced a dual approach using FID scores and expert radiologist assessments. Their findings revealed that although synthetic X-rays sometimes appeared realistic, they often lacked clinical reliability. Mertes et al. [ 5 ] focused on explainability, and their user study revealed significant improvements in mental models, explanation satisfaction, trust, emotions, and self-efficacy compared to traditional saliency-based explanation methods like LIME and LRP. This work highlighted the importance of considering human factors in evaluating synthetic medical images. Comparative evaluations have increasingly favored DMs. Muller-Franzes et al. [ 6 ] showed that DMs outperformed GANs on multiple datasets, including CheXpert, with markedly better FID, precision, and recall. Their Medfusion study established DMs as the new standard for diverse, high-quality generation across pathologies. Domain adaptation remains a challenge. Chambon et al. [ 1 ] addressed this by introducing RoentGen, a diffusion model fine-tuned on chest X-rays and radiology reports. They demonstrated improved image quality and representation of specific diseases, along with modest boosts in downstream task performance when using synthetic training data. Despite progress, gaps persist. Metrics like FID lack clinical interpretability, multi-disease synthesis remains difficult, and evaluation protocols are not standardized. Moreover, few studies analyze how radiologists discern image authenticity or examine which pathologies are well represented. Our study addresses these gaps through a dual radiologist-led evaluation of GAN and DM-generated X-rays, focusing on clinical realism, conditional accuracy, and visual patterns that inform human judgment. Figure 1: Interface of Task 1 concerning realism. The user is asked to identify which image is synthetic when displayed side-by-side with a real one. Five possible answers are displayed to capture different levels of confidence. Also, the user should select which aspect of the image influenced their decision. 3 Materials and Methods As mentioned in the introduction, it is essential to obtain radiologists’ evaluations to ensure the validity of our generated images. Thus, we compiled a chest X-ray dataset that included real and synthetic images. Then, we engaged a group of three radiologists in two specific tasks. The first task focused on realism: radiologists were asked to identify the synthetic image when paired with a real one and to detail the particular features observed in each image. The second task involved assessing whether the image accurately represented the indicated condition. After that, we collected, processed, and analyzed the radiologists’ labels to gauge the realism and conditional accuracy of the generated images. Figure 2: Web interface: Task 2 concerning conditionality. The user is asked whether the given label correctly classifies the displayed image. Both real and synthetic images are evaluated. The user should answer according to a scale of five levels of agreement. 3.1 Dataset Construction We created a dataset of 480 PA chest X-rays: 160 real images from the MIMIC-CXR [ 4 ] along with the Chest ImaGenome Dataset [ 10 ] , and 320 synthetic images (160 from a StyleGAN2 model [ 9 ] and 160 from a text-conditioned Diffusion model for chest X-rays, RoentGen [ 1 ] ). The text to condition the X-ray generation on the presence of a finding was simply the name of the abnormality, e.g., Pleural Effusion . To condition the generation for the absence of an abnormality, the input text was No followed by the abnormality, e.g., No pleural effusion . Each image was associated with one of four abnormalities: Atelectasis ( AT ), Lung Opacity ( LO ), Pleural Effusion ( PE ), or Enlarged Cardiac Silhouette ( ECS ), marked as both present or absent. GAN images were generated with class-specific binary models; DM images were produced using RoentGen with text prompts. 3.2 Evaluation Tasks

Read the complete text