Rethinking FID Through the Geometry of the Reference Dataset

Yunghee Lee, Byeonghyun Pak · 2026

Read Rethinking FID Through the Geometry of the Reference Dataset by Yunghee Lee, Byeonghyun Pak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29335.txt.

Introduction

Yunghee Lee and Byeonghyun Pak rethink Frechet Inception Distance through reference-dataset geometry. FID is widely used to judge image generators, yet lower FID does not always mean better samples. Across six datasets, density and effective rank explain much of how FID changes as sample quality improves. Concentrated datasets yield more favorable FID trends, whereas dispersed datasets can make FID worsen despite better samples. Such metrics should thus be read together with reference-dataset geometry.

Catalog record

CREATORYunghee Lee, Byeonghyun Pak
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29403
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29335

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking FID Through the Geometry of the Reference Dataset Yunghee Lee, Byeonghyun Pak Abstract Fréchet Inception Distance (FID) is widely used to evaluate image generators, yet lower FID does not always correspond to better sample quality. We show that this mismatch depends in part on the geometry of the reference dataset. In a controlled study across six datasets, distributional density and effective rank significantly explain how FID changes as sample quality improves. Concentrated datasets tend to yield more favorable FID trends, whereas more dispersed datasets can make FID worsen despite better samples. Attribution to precision and recall and ablations with alternative feature spaces and distances support the same conclusion. These results suggest that distributional metrics should be interpreted together with the geometry of the reference dataset for more reliable benchmarking. 1 Introduction Recent progress in image generation ( 28 ; 24 ; 26 ) has been accompanied by the widespread adoption of Fréchet Inception Distance (FID) ( 7 ) as a standard metric for evaluating generative models. FID estimates the discrepancy between the distribution of Inception-v3 ( 33 ) features extracted from real images and that of generated images, and lower FID has therefore been widely interpreted as evidence of a better generator. In practice, FID has become not only a reporting convention but also a benchmark target that shapes model development, hyperparameter tuning, and claims of progress across the literature ( 20 ; 11 ) . This central role implicitly assumes that improvements in FID correspond to improvements in the qualities that matter for downstream or real-world use. However, a growing body of evidence suggests that this assumption is unreliable. 2 reported that allocating more computation per sample produces visibly sharper images, yet worsens FID on the COCO dataset ( 18 ) . 15 showed that tuning a generation hyperparameter to minimize FID can yield the worst per-sample quality as measured by ImageReward ( 36 ) . Likewise, 8 demonstrated cases where stronger image distortions lead to better FID scores. If FID simply measured generator quality, such adjustments should not consistently degrade the metric while improving perceptual quality, or vice versa. These contradictions raise a broader concern: current benchmarking practice may fail to provide reliable or predictive insight into model behavior in realistic settings, even when the reported metric appears rigorous and standardized. Figure 1 : Distributional metrics depend on four components. Prior work has studied the generator, feature extractor, and estimator; we investigate the reference dataset. Prior work has mainly sought explanations in the fragility of Inception-v3 ( 12 ; 8 ; 23 ) and in the limitations of the Fréchet distance itself ( 3 ; 14 ) . While these analyses are important, they leave one basic aspect of the metric underexplored: the role of the reference dataset . Because FID is defined as a distance between the generated distribution and a chosen real-image reference distribution, its behavior necessarily depends on the structure of that reference set. Yet the choice of reference dataset is rarely justified, and is often inherited from prior benchmarks ( 4 ; 22 ) . A dataset such as the CelebA-HQ dataset ( 9 ; 35 ) concentrates around a dominant visual mode, whereas the COCO dataset ( 18 ) spans a far broader and more heterogeneous set of semantic categories; these datasets therefore occupy markedly different regions of feature space. This observation motivates a fundamental question for theory-informed benchmarking: which characteristics of the reference dataset determine the properties that FID favors? In this paper, we investigate how the geometry of a reference dataset shapes the behavior of FID. We characterize each dataset using two geometric properties: distributional density ( 19 ) and effective rank ( 27 ) . We then test whether these properties moderate the effect of generated sample quality on FID. Our results show that distributional density is a significant moderator of the quality-FID relationship. For dispersed datasets, FID can worsen even when per-sample quality is improved , offering a potential explanation for contradictions noted in earlier studies ( 31 ) . We further decompose FID into precision and recall ( 13 ) and find that, for such datasets, FID is more strongly correlated with recall, which can improve even as sample quality worsens. This dependence on dataset remains significant when replacing Inception-v3 with DINOv2 ( 21 ) and when replacing Fréchet distance with MMD ( 6 ) as in KID ( 1 ) . Therefore we argue that distributional metrics should be interpreted together with reference dataset geometry , and that understanding the reference dataset is essential for reliable benchmarking of generative models. In summary, our contributions are as follows: • We present a controlled empirical study showing that reference dataset geometry, quantified by distributional density and effective rank, significantly moderates FID. • We perform ablations across alternative feature spaces (DINOv2) and alternative kernel-based metrics (KID), showing that the observed effect is not specific to Inception-v3 features or the Fréchet estimator. • We provide practical guidance for evaluating distributional metrics: use FID with concentrated datasets, or report it together with the geometric descriptors. 2 Method 2.1 Geometric Properties of Dataset We summarize each reference dataset with two scalar descriptors of its feature distribution. Both descriptors are computed in the same feature space as the metric. Distributional Density. Distributional density measures how tightly the reference distribution concentrates around its own points. A high value means the samples are clustered and close to each other; a low value means they are scattered and far away. A standard estimator of concentration is the kNN density ( 19 ) , \hat{p}(x)\propto d_{k}(x)^{-D} , where d_{k}(x) denotes the Euclidean distance from x to its k -th nearest neighbor and D is the feature dimension. However, in the Inception-v3 feature space with D=2048 , the exponent makes \hat{p} impractical as a dataset-level summary, as its values span tens of orders of magnitude across datasets. To obtain a more stable descriptor, we take the logarithm and then average over the reference set, reducing the pointwise estimate to a single scalar. The resulting descriptor, which we call the mean kNN log-density , is defined as \langle-\log d_{k}\rangle=\frac{1}{n}\sum_{i=1}^{n}-\log d_{k}(x_{i}), (1) where n is the number of reference points. We use k=80 throughout this paper. Effective Rank. Effective rank measures how many linear directions the feature distribution spreads over. A high value means the support occupies many principal components of the feature space; a low value means it collapses onto a few. The effective rank of a matrix A is ( 27 ) \mathrm{erank}(A)=\exp\left(H(\bm{\sigma}/\|\bm{\sigma}\|_{1})\right), (2) where \bm{\sigma} is the vector of singular values of A , \|\bm{\sigma}\|_{1} is the L1 norm of \bm{\sigma} , and H(\bm{p})=-\sum_{k}p_{k}\log p_{k} is the Shannon entropy ( 29 ) . \mathrm{erank}(A) equals \mathrm{rank}(A) when nonzero singular values are all equal, which gives it the interpretation of weighted dimensionality. We use \mathrm{erank}(A) as the descriptor where A is the centered feature matrix. Table 1 : Distributional density \langle-\log d_{k}\rangle and effective rank \mathrm{erank}(A) of the six reference datasets in the Inception-v3 feature space. Dataset \langle-\log d_{k}\rangle \mathrm{erank}(A) FFHQ ( 10 ) -2.48 1243 CelebA-HQ ( 9 ) -2.36 1220 MJHQ-30K ( 17 ) -2.74 1341 ImageNet ( 5 ) -2.68 1431 Flickr30K ( 37 )

Read the complete text