Representation Fréchet Loss for Visual Generation

Jiawei Yang Zhengyang Geng Xuan Ju Yonglong Tian Yue Wang, [0.5em] USC · 2026

Read Representation Fréchet Loss for Visual Generation by Jiawei Yang Zhengyang Geng Xuan Ju Yonglong Tian Yue Wang, [0.5em] USC (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-28190.txt.

Introduction

Yang and colleagues show Fréchet distance, long deemed impractical as a training objective, can be optimised in representation space. Their FD-loss decouples the population size used to estimate the distance from the batch size used for gradients. Post-training with FD-loss improves visual quality across representation spaces, turns multi-step generators into strong single-pass generators without distillation or adversarial training, and motivates a multi-representation metric, since FID can misrank quality.

Catalog record

CREATORJiawei Yang Zhengyang Geng Xuan Ju Yonglong Tian Yue Wang, [0.5em] USC
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79022
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.28190

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Representation Fréchet Loss for Visual Generation Jiawei Yang Zhengyang Geng Xuan Ju Yonglong Tian Yue Wang, [0.5em] USC Abstract We show that Fréchet Distance (FD), long considered impractical as a training objective, can in fact be effectively optimized in the representation space. Our idea is simple: decouple the population size for FD estimation ( e.g . , 50k) from the batch size for gradient computation ( e.g . , 1024). We term this approach FD-loss . Optimizing FD-loss reveals several surprising findings. First, post-training a base generator with FD-loss in different representation spaces consistently improves visual quality. Under the Inception feature space, a one-step generator achieves 0.72 FID on ImageNet 256 × 256 256\times 256 . Second, the same FD-loss repurposes multi-step generators into strong one-step generators without teacher distillation, adversarial training or per-sample targets. Third, FID can misrank visual quality: modern representations can yield better samples despite worse Inception FID. This motivates FDr k \text{FDr}^{k} , a multi-representation metric. We hope this work will encourage further exploration of distributional distances in diverse representation spaces as both training objectives and evaluation metrics for generative models. Code and checkpoints are available at https://github.com/Jiawei-Yang/FD-loss . pMF-H (1-NFE) pMF-H + FD-loss (1-NFE) JiT-H (1-NFE) JiT-H + FD-loss (1-NFE) Figure 1 : One-step samples on ImageNet 256 \times 256, before and after FD-loss post-training. Left : samples from the base generators. Right : samples after post-training with FD-loss . Top two rows : pMF-H lu2026one , a one-step generator. Bottom : JiT-H li2025back , a multi-step generator. All models generate in a single network evaluation (1-NFE). FD-loss improves existing one-step models and can repurpose multi-step models into one-step generators. See uncurated samples in Appendix E . 1 Introduction Generator θ Noise Batch \displaystyle B \cdots \cdots \nabla_{\theta} Population \displaystyle N FD-loss \nabla_{\theta} Figure 2 : FD-loss . The generator produces \displaystyle B images per step; their features are accumulated into a large population via a queue or EMA. This decouples the population size \displaystyle N for reliable FD estimation from the batch size \displaystyle B for gradient computation. The Fréchet Inception Distance (FID) heusel2017gans has been the de facto metric for evaluating image generation, especially on academic benchmarks such as ImageNet deng2009imagenet . For nearly a decade, the community has been collectively performing “gradient descent” on this single metric to push the state of the art. But this “gradient descent” has always been indirect : FID serves only as an evaluator, not as a loss. In this paper, we ask: can FID be optimized directly as a training loss—and what does that reveal? In principle, this is straightforward: every term in Fréchet Distance (FD) is differentiable and nothing in its definition restricts it to evaluation. In practice, however, this has been widely considered impractical. Computing FID requires a large population of samples ( e.g . , 50k) to estimate distributional statistics. Using it as a training loss would further demand gradients through all of them at every step, which is prohibitive. Prior work has therefore only explored FD as a loss under restricted settings, e.g . , estimating FD from a small batch mathiasen2020backpropagating ; doan2020image . Our experiments confirm that optimizing batch-wise FD degrades base generators (Tab. ). In this work, we show that FD can in fact be optimized directly at scale . Our idea is simple (Fig. 2 ): decouple the population size used for FD estimation from the batch size used for gradient computation. We call this approach FD-loss and realize it in two ways. The first maintains an online queue of features from recently generated samples and computes FD over the full queue ( e.g . , 50k) while back-propagating only through the current batch. The second maintains exponential moving averages (EMA) of the first and second moments of features and restricts gradients to the current batch. Both work well in practice. This one approach unlocks several surprising findings. First, FD-loss is a strong post-training objective for visual generators. We fine-tune a pre-trained generator with FD-loss using only pre-computed feature statistics---means and covariances of real data under one or more representation backbones. 1 1 1 Real data is used only once, offline, to compute these reference statistics. No real images are seen during post-training. Across pixel-space and latent-space generator families lu2026one ; geng2025improved ; li2025back , model sizes, and image resolutions, this consistently improves visual quality. Under Inception szegedy2016rethinking , FD-loss drives a one-step generator lu2026one to an FID of 0.72 on ImageNet 256\times 256 (Tab. 4 ). The same recipe, applied under modern representations liu2022convnet ; he2022masked ; oquab2024dinov2 ; radford2021learning ; tschannen2025siglip , yields even stronger perceptual quality (Fig. 4 , Tab. 3 ). Second, FD-loss serves as a simple distribution-matching objective. We show that a pre-trained multi-step generator can be repurposed into a one-step generator by post-training it with FD-loss (from around 300 FID to 0.72). This repurposing works without teacher distillation, adversarial training, or per-sample targets, suggesting that the representation-space FD is a useful distribution-level objective rather than only an evaluation metric. We validate this on both class-conditioned models (Fig. 1 , bottom) and text-conditioned models (Fig. 7 ). Third, FD-loss provides a diagnostic lens. Having models trained by FD-loss with different representations, we can test whether the best-FID model is also the perceptually best. Often, it is not : models optimized with modern representations achieve better visual quality while scoring worse under FID (Tab. , Fig. 4 ). More broadly, this connects to a paradox in the field (Fig. 3 ): state-of-the-art generators already surpass real validation images under FID, yet their outputs remain clearly distinguishable from real images. We therefore report \text{FDr}^{k} , a normalized FD ratio averaged over k feature spaces (Eq. 8 ), as a more representation-diverse automatic metric. As shown in Figure 3 , FID suggests the “ImageNet generation” problem is nearly solved, yet \text{FDr}^{k} reveals that significant quality gaps remain. Post-training with FD-loss narrows this gap, achieving an \text{FDr}^{6} of 1.89. 2024 2025 2026 0.5 0.7 1.0 1.5 2.0 3.0 Val. ref = 1.68 REPA DDT REPA-E RAE BAR SiT VAR FlowAR MAR DeTok iMF JiT PixNerd Drift pMF iMF ∗ pMF ∗ JiT ∗ “Solved”? FID \downarrow (log-scale) 2024 2025 2026 1.0 2.0 3.0 5.0 7.0 10.0 Val. ref = 1.0 REPA DDT REPA-E RAE BAR SiT VAR FlowAR MAR DeTok iMF JiT PixNerd Drift pMF iMF ∗ pMF ∗ JiT ∗ Not yet \text{FDr}^{6} \downarrow (log-scale) \bullet previous works \blacklozenge FD-loss (ours) ∗ ​: post-trained by FD-loss with SigLIP2 tschannen2025siglip , Inception szegedy2016rethinking , and MAE he2022masked for 100 epochs. Figure 3 : Is ImageNet generation “solved”? Left : FID over time. Recent methods surpass the real validation set images (red dashed) in terms of FID. Right : \text{FDr}^{6} (Eq. 8 ), which averages normalized Fréchet Distance ratios across six representation spaces. Under this metric, even the strongest existing methods remain far from the validation images, indicating that FID alone masks significant quality gaps. Each method is shown at its largest publicly available model size. Post-training with FD-loss ( \blacklozenge ) improves both metrics; see Table 4 for numerical results. Taken together, our findings reposition FD in generative modeling. FD has long lived on one side of

Read the complete text