A Pragmatic Note on Evaluating Generative Models with Fréchet Inception Distance for Retinal Image Synthesis

Yuli Wu 1,3, Fucheng Liu 1, Rüveyda Yilmaz 1, Henning Konermann 1, Peter Walter 1,2, Johannes Stegmaier 1,3 · 2025

Read A Pragmatic Note on Evaluating Generative Models with Fréchet Inception Distance for Retinal Image Synthesis by Yuli Wu 1,3, Fucheng Liu 1, Rüveyda Yilmaz 1, Henning Konermann 1, Peter Walter 1,2, Johannes Stegmaier 1,3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17160.txt.

Introduction

Yuli Wu and five colleagues question a popular score for judging AI-made medical images. The Fréchet Inception Distance, or FID, compares synthetic pictures with real ones using a network trained on everyday photos. It is the default yardstick for image generators, including those that fake retinal scans.

For medicine, the real goal is usually richer training data for diagnosis tools. The authors show cases in eye imaging, fundus photos and coherence scans, where FID and its cousins point the wrong way. A set can score well yet fail to help a classifier or segmentation model learn. They urge judging generators by the downstream task instead and warn other imaging fields of the same trap.

Catalog record

CREATORYuli Wu 1,3, Fucheng Liu 1, Rüveyda Yilmaz 1, Henning Konermann 1, Peter Walter 1,2, Johannes Stegmaier 1,3
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-17160

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Pragmatic Note on Evaluating Generative Models with Fréchet Inception Distance for Retinal Image Synthesis Yuli Wu 1,3, Fucheng Liu 1, Rüveyda Yilmaz 1, Henning Konermann 1, Peter Walter 1,2, Johannes Stegmaier 1,3 Abstract Fréchet Inception Distance (FID), computed with an ImageNet pretrained Inception-v3 network, is widely used as a state-of-the-art evaluation metric for generative models. It assumes that feature vectors from Inception-v3 follow a multivariate Gaussian distribution and calculates the 2-Wasserstein distance based on their means and covariances. While FID effectively measures how closely synthetic data match real data in many image synthesis tasks, the primary goal in biomedical generative models is often to enrich training datasets ideally with corresponding annotations. For this purpose, the gold standard for evaluating generative models is to incorporate synthetic data into downstream task training, such as classification and segmentation, to pragmatically assess its performance. In this paper, we examine cases from retinal imaging modalities, including color fundus photography and optical coherence tomography, where FID and its related metrics misalign with task-specific evaluation goals in classification and segmentation. We highlight the limitations of using various metrics, represented by FID and its variants, as evaluation criteria for these applications and address their potential caveats in broader biomedical imaging modalities and downstream tasks. \midlauthor 1 RWTH Aachen University, Aachen, Germany 2 Uniklinik RWTH Aachen, Aachen, Germany 3 Heinrich Heine University Düsseldorf, Düsseldorf, Germany renddoc † † volume: – nnn † † year: 2026 † † workshop: Full Paper – MIDL 2026 † † editors: Accepted for publication at MIDL 2026 keywords FID, Generative models, Retinal imaging. 1 Introduction Deep generative models, particularly generative adversarial networks (GANs) Goodfellow et al. (2014) , by adversarial training of the generator and discriminator, and diffusion models Ho et al. (2020) , by iteratively refining noise into structured images, have demonstrated significant potential in 2D and 3D biomedical image synthesis, as shown in studies such as Eschweiler et al. (2024) ; Han et al. (2020) ; Müller-Franzes et al. (2023) ; Wu et al. (2024) ; Yilmaz et al. (2024) ; Yilmaz et al. (2025) . By learning from real biomedical data, these models can generate realistic synthetic images, helping to address challenges like limited data availability and data privacy concerns. Correspondingly, various studies have focused on generating fully annotated images to support downstream task training, such as classification and segmentation Park et al. (2019) ; Zhang et al. (2023) . This can be achieved by conditioning generative models on different types of input, such as text, images, or sketches. Furthermore, several approaches control the diffusion process using structured guidance, where segmentation maps are used to steer image synthesis Eschweiler et al. (2024) ; Park et al. (2019) ; Yilmaz et al. (2024) ; Zhang et al. (2023) . The resulting image-annotation pairs are then pragmatically used for training downstream tasks. However, ensuring that synthetic images are both realistic and useful for downstream tasks remains a critical challenge Theis et al. (2016) ; Konz et al. (2024a) . In ophthalmic imaging particularly, generative models have enabled explainable classification with GAN-based styles Lang et al. (2021) , lesion-map conditioned diabetic retinopathy fundus generation using GANs Hou (2023) , diffusion-based fundus image generation Müller-Franzes et al. (2023) , OCT image–label generation for layer segmentation Wu et al. (2024) ; Huang et al. (2024) , counterfactual retinal synthesis Ilanchezian et al. (2025) , and controllable fundus synthesis via disentangled representations Müller et al. (2025) . A key open question is whether feature-distance metrics (see section 2 ), which quantify distributional similarity in the embedding space of a fixed pretrained model, are reliable proxies for the practical utility of synthetic data in downstream training. In this work, we study data enrichment and ask: (i) do different feature-distance metrics yield consistent rankings of generative model variants, as quantified by Kendall’s \tau rank correlation; and (ii) do these rankings align with downstream performance in classification and segmentation when synthetic samples are added to the training set? To answer these questions, we evaluate multiple generators on retinal fundus and OCT synthesis, compute seven representative metrics spanning different distances and feature extractors, and compare their rank orderings with downstream task performance on held-out test data. Figure 1: Overview of the generative model pipeline for data enrichment exemplified with retinal images, including generation, evaluation, and downstream task prediction. The discrepancy between generative evaluation metrics and downstream performance is investigated. As shown in fig:pipeline, we illustrate a common scenario using generative models, demonstrated with two retinal image modalities for data enrichment. On one hand, a synthetic dataset is generated with optional annotations and evaluated using various metrics, primarily FID and its variants ( 2 ). On the other hand, the composite dataset containing a mixture of real and synthetic data is applied to a downstream task, where the trained model predicts the desired outputs on the test set ( 3 ). In 4 , we demonstrate that widely used generative evaluation metrics fail to align with downstream performance and draw a pragmatic note on the unreliability of such feature-distance metrics when assessing generative models for data enrichment in the context of a utilitarian downstream task. 2 Generative Evaluation Metrics 2.1 Fréchet Inception Distance Fréchet Inception Distance (FID) Fréchet (1957) ; Heusel et al. (2017) is the de facto standard metric for assessing the perceptual quality of generated images. It compares the feature distributions of generated and real images using an ImageNet Deng et al. (2009) pretrained Inception-v3 model Szegedy et al. (2016) and the Fréchet distance (equivalent to the 2-Wasserstein distance Vaserstein (1969) ; Peyré and Cuturi (2019) ), which measures the difference between two probability distributions: p_{r} , the distribution of real data, and p_{g} , the distribution of generated data. Assuming Gaussian distributions with means \mu_{r},\mu_{g} and covariances \Sigma_{r},\Sigma_{g} , the squared Fréchet distance (FD) is derived as: D_{\text{Fréchet}}^{2}(p_{r},p_{g})=\|\mu_{r}-\mu_{g}\|^{2}+\text{tr}\left(\Sigma_{r}+\Sigma_{g}-2(\Sigma_{r}\Sigma_{g})^{1/2}\right), (1) where \|\mu_{r}-\mu_{g}\|^{2} is the squared Euclidean distance between means, and \text{tr}(\cdot) is the trace of a matrix. A lower FID score indicates that the generated images more closely resemble the real images in terms of perceptual similarity. It is important to note that the closed-form expression in ( 1 ) is not limited to the multivariate Gaussian distribution, but also applies to any two distributions in \mathbb{R}^{n} within the family of elliptically contoured distributions Dowson and Landau (1982) ; Peyré and Cuturi (2019) , expressed as: f({x};\mu,A)=\mathrm{const.}\times\frac{1}{|A|^{1/2}}\,g\left(({x}-{\mu})^{\top}A^{-1}({x}-{\mu})\right), (2) where A is a positive definite matrix and g(z) is a non-negative function defined on the positive real axis ( r ), satisfying 0<\int_{0}^{\infty}r^{n/2-1}g(r)dr<\infty , which characterizes the specific distribution (e.g., for the Gaussian distribution, g(z)=\exp(-z/2) ). Therefore, merely verifying the non-Gaussianity of the feature distributions does not undermine the reliability of FID (cf. Jayasumana et al. (2024) ). By substituting the feature extractor in the FID calculation with alternative models or applying different distance measures for feature comparison, we can explore alternative approaches to evaluate generative models. We refer to this class of generative evaluation metrics as feature-distance metrics and present variants of FID, categorized by different distance measures in 2.2 and feature extractors in 2.3 . 2.2 Distance and Divergence Variants Besides the Fréchet distance, Maximum Mean Discrepancy (MMD) Gretton et al. (2006) is another key distance metric for

Read the complete text