Fair Benchmarking of Emerging One-Step Generative Models Against Multistep Diffusion and Flow Models
Advaith Ravishankar ⋆, Serena Liu ⋆, Mingyang Wang ⋆, Todd Zhou, Jeffrey Zhou, Arnav Sharma, Ziling Hu, Léopold Das, Abdulaziz Sobirov, Faizaan Siddique, Freddy Yu, Seungjoo Baek, Yan Luo †, Mengyu Wang † · 2026
Read Fair Benchmarking of Emerging One-Step Generative Models Against Multistep Diffusion and Flow Models by Advaith Ravishankar ⋆, Serena Liu ⋆, Mingyang Wang ⋆, Todd Zhou, Jeffrey Zhou, Arnav Sharma, Ziling Hu, Léopold Das, Abdulaziz Sobirov, Faizaan Siddique, Freddy Yu, Seungjoo Baek, Yan Luo †, Mengyu Wang † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14186.txt.
Introduction
Ravishankar and co-authors benchmark one-step image generators against multi-step diffusion and flow models under matched settings. Past comparisons mix step counts and guidance settings, which can move scores in opposed ways. Eight models are tested on ImageNet and two further sets, including a new proofread out-of-distribution set. Scores cover FID, Inception, CLIP, and Pick metrics. The team shows that chasing FID alone can mislead in few-step regimes, since guidance tweaks may lift FID while hurting text fit.
Catalog record
| CREATOR | Advaith Ravishankar ⋆, Serena Liu ⋆, Mingyang Wang ⋆, Todd Zhou, Jeffrey Zhou, Arnav Sharma, Ziling Hu, Léopold Das, Abdulaziz Sobirov, Faizaan Siddique, Freddy Yu, Seungjoo Baek, Yan Luo †, Mengyu Wang † |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51905 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14186
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fair Benchmarking of Emerging One-Step Generative Models Against Multistep Diffusion and Flow Models Advaith Ravishankar ⋆, Serena Liu ⋆, Mingyang Wang ⋆, Todd Zhou, Jeffrey Zhou, Arnav Sharma, Ziling Hu, Léopold Das, Abdulaziz Sobirov, Faizaan Siddique, Freddy Yu, Seungjoo Baek, Yan Luo †, Mengyu Wang † Abstract State-of-the-art text-to-image models produce high-quality images, but inference remains expensive as generation requires several sequential ODE or denoising steps. Native one-step models aim to reduce this cost by mapping noise to an image in a single step, yet fair comparisons to multi-step systems are difficult because studies use mismatched sampling steps and different classifier-free guidance (CFG) settings, where CFG can shift FID, Inception Score, and CLIP-based alignment in opposing directions. It is also unclear how well one-step models scale to multi-step inference, and there is limited standardized out-of-distribution evaluation for label-ID-conditioned generators beyond ImageNet. To address this, We benchmark eight models spanning one-step flows (MeanFlow, Improved MeanFlow, SoFlow), multi-step baselines (RAE, Scale-RAE), and established systems (SiT, Stable Diffusion 3.5, FLUX.1) under a controlled class-conditional protocol on ImageNet validation, ImageNetV2, and reLAIONet, our new proofread out-of-distribution dataset aligned to ImageNet label IDs. Using FID, Inception Score, CLIP Score, and Pick Score, we show that FID-focused model development and CFG selection can be misleading in few-step regimes, where guidance changes can improve FID while degrading text-image alignment and human preference signals and worsening perceived quality. We further show that leading one-step models benefit from step scaling and become substantially more competitive under multi-step inference, although they still exhibit characteristic local distortions. To capture these tradeoffs, we introduce MinMax Harmonic Mean (MMHM), a composite proxy over all four metrics that stabilizes hyperparameter selection across guidance and step sweeps. The code for reproducing the benchmarks is made publicly available at github.com/Harvard-AI-and-Robotics-Lab/FairBenchmarkingFlow . 1 Introduction Recent progress in image generation has been driven by diffusion and flow-based models, producing highly realistic samples but requiring several sequential denoising or ODE steps at inference. This multi-step sampling makes state-of-the-art systems expensive to deploy at scale, since generating a single image can require dozens of forward passes through billion-parameter backbones. Motivated by this compute bottleneck, a new line of work targets native one-step generation, where a single model evaluation maps noise directly to an image. While early efforts relied on distillation, several methods now report strong one-step results without a teacher, including MeanFlow [ 6 ] , Improved MeanFlow (iMF) [ 7 ] , and SoFlow [ 15 ] . Despite this rapid progress, comparing one-step models to established multi-step systems remains nontrivial. First, models are often reported under heterogeneous inference settings: one-step flow models are typically trained and evaluated under ImageNet label-ID conditioning, while state-of-the-art systems such as Stable Diffusion 3.5 and FLUX.1 are trained for text conditioning. Second, reported results depend on inference heuristics and hyperparameters, especially classifier-free guidance (CFG), which can substantially alter both quantitative metrics and qualitative appearance. Third, the field often centers evaluation around Fréchet Inception Distance (FID), which measures sample quality in the InceptionNetv3 feature space but does not directly capture text-image alignment or human preference. Together, these factors make it unclear how competitive one-step models are under matched guidance and step settings, and whether optimizing FID alone yields the best visual quality. In this work, we present a controlled benchmark of emerging one-step flow models against established diffusion and flow-based systems under consistent inference protocols. We evaluate eight approaches spanning native one-step models (MeanFlow, iMF, SoFlow), established and emerging flow baselines (Scalable Interpolant Transformer (SiT) [ 16 ] , Representation Autoencoder (RAE) [ 25 ] , Scale-RAE [ 24 ] ), and state-of-the-art text-to-image systems (Stable Diffusion 3.5 Large [ 5 ] , FLUX.1 [ 1 ] ). For each method, we compare matched guidance (CFG =7 where applicable) and each model’s reported native guidance setting. We also evaluate both one-step and multi-step inference (1 vs. 25 steps) to measure how one-step models behave when iterated and how multi-step systems change under reduced step budgets. To test generalization beyond the ImageNet family while preserving label-ID conditioning, we introduce reLAIONet , a new proofread evaluation dataset that maps web-sourced images to ImageNet classes and is manually filtered to remove mislabeled and ambiguous samples. reLAIONet provides a larger, more diverse out-of-distribution test set than ImageNetV2 while retaining the class-conditional interface required by ImageNet-trained generators. Finally, we study how metric choice shapes conclusions about generative quality. First, we find that minimizing FID alone can systematically trade off against Inception Score, CLIP Score, and Pick Score, while selecting configurations that balance all four metrics via MinMax Harmonic Mean (MMHM) yields more consistent performance and better qualitative samples. This pattern holds across all datasets, suggesting it is inherent to single-metric tuning rather than dataset-specific. Second, using MMHM-based ranking rather than best-FID settings, the strongest native one-step models narrow the gap to state-of-the-art baselines (SD3.5 Large and FLUX.1) under a multi-step class-conditional protocol quantitatively, indicating one-step formulations can scale under controlled inference settings. However, one-step models still lag in qualitative fidelity as they exhibit noticeable distortions with generated images. An important next step is testing whether these gains translate to text-conditioned generation at scale, since our study focuses on class conditioning. Third, MMHM is not a definitive target objective; rather, it highlights that multi-metric selection yields more reliable conclusions and outputs than any single metric. Designing a principled, optimization-ready objective that better tracks human-perceived quality remains an open direction for future work. Figure 1: Left: Example images from ImageNet, ImageNetV2, and reLAIONet for the same label IDs, illustrating the distribution shift of reLAIONet. Middle/Right: Qualitative samples from Scuba Diver and Sports Car classes, where lower FID still exhibit visible distortions (e.g., faces and local structure). The reported FID is a model-level score computed over the full evaluation set, so all samples from a given model share the same value; additional examples are in the supplement. 2 Related Work 2.1 Diffusion and Flow Models Since 2020, diffusion-based approaches have dominated visual generation. Denoising diffusion probabilistic models (DDPMs) initiated this trend by learning to reverse a fixed noising process through several sequential stochastic denoising steps [ 9 ] . Since DDPMs, diffusion models have been improved in two major ways. First, latent diffusion moved the diffusion process into a compressed autoencoder latent space, enabling higher-resolution generation with reduced compute and flexible cross-attention conditioning [ 19 ] . Second, denoising diffusion implicit models (DDIM) replaced the stochastic reverse-process formulation with a deterministic ODE-style sampler, allowing stable trajectories and fewer effective timesteps via numerical integration [ 23 ] . This ODE view helped motivate flow-based reformulations of multi-step generation: rather than predicting noise, flow models learn a velocity field that transports samples from noise to data. Flow Matching formalized this by regressing a vector field along probability paths between noise and images [ 12 ] , and Rectified Flow improved few-step sampling by encouraging straighter trajectories and reducing discretization error [ 13 ] . Building on these ideas, SiT unified diffusion and flow objectives under a single interpolant framework and showed strong scalability with transformer backbones [ 16 ] . Today, state-of-the-art text-to-image systems increasingly adopt these flow-style designs, including Stable Diffusion 3.5 and FLUX.1 [ 5 , 1 ] . Despite these advances,