Enhanced Generative Model Evaluation with Clipped Density and Coverage

Nicolas Salvy, Hugues Talbot, Bertrand Thirion · 2025

Read Enhanced Generative Model Evaluation with Clipped Density and Coverage by Nicolas Salvy, Hugues Talbot, Bertrand Thirion (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01761.txt.

Introduction

Nicolas Salvy, Hugues Talbot, and Bertrand Thirion introduce two scores for judging generative models, in a 2025 paper. Such models make images and other samples, but quality is hard to score. Quality splits into fidelity, how real samples look, and coverage, how much variety they span. Current scores can be skewed by outliers and are hard to read.

Their scores clip each sample's contribution before averaging. Radii around samples are clipped as well for the fidelity score. That stops stray bad samples from dragging the totals. Scores fall in a straight line as the share of bad samples grows, tests showed. So a score can be read as a plain share of good samples. Trials on fake and real data beat existing methods on robustness and clarity.

Catalog record

CREATORNicolas Salvy, Hugues Talbot, Bertrand Thirion
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-01761

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhanced Generative Model Evaluation with Clipped Density and Coverage Nicolas Salvy, Hugues Talbot, Bertrand Thirion Abstract Although generative models have made remarkable progress in recent years, their use in critical applications has been hindered by their incapacity to reliably evaluate sample quality. Quality refers to at least two complementary concepts: fidelity and coverage. Current quality metrics often lack reliable, interpretable values due to an absence of calibration or insufficient robustness to outliers. To address these shortcomings, we introduce two novel metrics, Clipped Density and Clipped Coverage . By clipping individual sample contributions and, for fidelity, the radii of nearest neighbor balls, our metrics prevent out-of-distribution samples from biasing the aggregated values. Through analytical and empirical calibration, these metrics exhibit linear score degradation as the proportion of poor samples increases. Thus, they can be straightforwardly interpreted as equivalent proportions of good samples. Extensive experiments on synthetic and real-world datasets demonstrate that Clipped Density and Clipped Coverage outperform existing methods in terms of robustness, sensitivity, and interpretability for evaluating generative models. Figure 1 : Measuring the coverage of a mixture of good and bad samples (CIFAR-10) : This plot shows a collection of coverage metrics, relative to their maximal value, as a function of bad synthetic sample proportion. Ideally, scores should degrade linearly with increasing bad sample proportion. Only Clipped Coverage displays this desired linear behavior. 1 Introduction Generative models have achieved remarkable progress in recent years and are being actively explored in various fields, such as healthcare pinaya2022brain ; fernandez2024generating ; tudosiu2024realistic ; zhu2024diffusion ; bluethgen2024vision However, their deployment in high-stakes applications depends on the ability to reliably evaluate the quality of synthetic data, ensuring it can be trusted. This evaluation is inherently challenging, especially for high-dimensional data. The true underlying distributions of these data are often unknown and complex. They also do not conform to known parametric families. This makes computing support or density infeasible in practice. Current model evaluation often relies on metrics like the Fréchet Inception Distance (FID) heusel2017gans and FD-DINOv2 stein2024exposing for images. These provide a single, compound score to represent overall sample quality, making it impossible to determine whether poor performance stems from a lack of realism or variety sajjadi2018assessing . To address this issue, the quality of synthetic data can be broken down into at least two core concepts: fidelity and coverage. Fidelity assesses how similar each synthetic sample is to the input data naeem2020reliable . Conversely, coverage measures the extent to which synthetic samples represent the distribution of real data, taking into consideration how the rarity or commonness of real data is reflected in the synthetic samples. To determine whether generative models can be truly dependable, particularly in sensitive applications, the metrics used to evaluate them must be trustworthy. This means that they must be robust to outliers, sensitive to genuine deficiencies in the synthetic data, and provide interpretable scores. Crucially, beyond relative comparisons (i.e., knowing that one model is better than another), metrics for which absolute values are meaningful are critical to ensure that the synthetic data can be used in applications. Ideally, a score of x would indicate performance equivalent to having a proportion of x good samples and (1-x) bad ones. As shown in fig. 1 , current coverage metrics fail to achieve this. We show in fig. 4 that current fidelity metrics are similarly untrustworthy. In this paper, we introduce Clipped Density and Clipped Coverage , two novel metrics designed to overcome these limitations. Our contributions are: • Trustworthy Evaluation : Our metrics achieve robustness to outliers by clipping the contribution of individual samples to the aggregate score and, for Clipped Density , by limiting the radii of nearest-neighbor spheres used to measure the density. This prevents out of distribution samples from unduly influencing the evaluation while maintaining sensitivity to actual quality issues. • Interpretable Absolute Scores : Through empirical calibration in the case of Clipped Density and theoretical analysis in the case of Clipped Coverage , we ensure that the values degrade linearly with the proportion of bad samples. The paper is structured as follows: Section 2 provides background on existing metrics. Section 3 reviews related work. Section 4 introduces our Clipped Density and Clipped Coverage metrics. Section 5 details our experiments and results, and Section 6 discusses implications and limitations. 2 Background We consider a setting in which we are given N i.i.d. samples \{x^{r}_{i}\}_{i=1}^{N} from an unknown data (reference, or real) distribution p_{r} and M i.i.d. samples \{x^{s}_{j}\}_{j=1}^{M} generated from an unknown synthetic data distribution p_{s} . In this section, we review relevant metrics for evaluating generative models that aim to disentangle two aspects of synthetic data: fidelity (how realistic each synthetic sample is) and coverage (how well the synthetic samples populate the real density distribution). These metrics serve as the basis for our proposed improvements. Assuming a given support S^{r} of the real distribution and S^{s} for the synthetic distribution, Precision sajjadi2018assessing ; simon2019revisiting measures the proportion of synthetic samples that fall in S^{r} , while Recall sajjadi2018assessing ; simon2019revisiting measures the proportion of i.i.d. real samples that fall in S^{s} . \displaystyle\text{Precision}=\mathbb{P}_{p_{s}}[S^{r}]=\mathbb{P}_{p_{s}}[S^{r}\cap S^{s}] \displaystyle\text{Recall}=\mathbb{P}_{p_{r}}[S^{s}]=\mathbb{P}_{p_{r}}[S^{r}\cap S^{s}] (1) However, the underlying densities d^{r} and d^{s} and their support S^{r} and S^{s} are unknown, making such computation infeasible. In practice, improved Precision and Recall kynkaanniemi2019improved estimate these distributions through the union of balls centered on each observed sample, with a radius equal to the distance \text{NND}_{k} from each center to its k -Nearest Neighbor BURMAN1992132 . Here, \text{NND}_{k}^{r} denotes the distance to the k -nearest real data sample, while \text{NND}_{k}^{s} denotes the distance to the k -nearest synthetic data sample. iPrecision \displaystyle=\mathbb{P}_{p_{s}}[\hat{S^{r}}\cap S^{s}]=\frac{1}{M}\sum_{j=1}^{M}\mathbf{1}_{x^{s}_{j}\in\cup_{i=1}^{N}B(x^{r}_{i},\text{NND}_{k}^{r}(x^{r}_{i}))} (2) iRecall \displaystyle=\mathbb{P}_{p_{r}}[S^{r}\cap\hat{S^{s}}]=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}_{x^{r}_{i}\in\cup_{j=1}^{M}B(x^{s}_{j},\text{NND}_{k}^{s}(x^{s}_{j}))} (3) Yet, iPrecision and iRecall are strongly biased by out-of-distribution samples naeem2020reliable . Real outliers, being far from their nearest neighbors, create large balls and bias the estimation of {S^{r}} . Similarly, bad samples in the synthetic data compromise the approximation of the synthetic support naeem2020reliable . Density and Coverage naeem2020reliable aim to alleviate this issue. Instead of making a binary decision on whether a synthetic sample lies within the approximated real manifold, Density counts the number of real balls a synthetic sample falls into, normalized by k . The rationale is that outliers with large balls only contribute \frac{1}{k} to the fidelity of a given synthetic sample, thus having a lesser contribution to the overall score than in iPrecision , where the sample fidelity would be 1 naeem2020reliable . With the definition given below, the Density metric represents the average relative density d^{s}/d^{r} . In high-dimensional spaces where measuring density is impractical, Density instead examines balls centered around observed data points, providing a relative measure of how many synthetic points fall within these balls normalized by their mass k . To avoid estimation biases when approximating the synthetic distribution that may contain many bad samples, additional considerations are needed: Coverage adopts a distinct perspective naeem2020reliable . Instead of counting the proportion of real samples within at least one synthetic ball, Coverage calculates the proportion of real samples that have at least one synthetic sample within their ball: \text{Coverage}=\mathbb{P}_{p_{r}}[\hat{S^{r}}\cap S^{s}] . It is the proportion of real samples that are covered . \displaystyle\text{Density}=\frac{1}{kM}\sum_{j=1}^{M}\sum_{i=1}^{N}\mathbf{1}_{x^{s}_{j}\in B(x^{r}_{i},\text{NND}_{k}^{r}(x^{r}_{i}))} \displaystyle\text{Coverage}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}_{\exists j,x^{s}_{j}\in B(x^{r}_{i},\text{NND}^{r}_{k}(x^{r}_{i}))} (4) Although Density is less

Read the complete text