Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization

Kyle Sargent, Kyle Hsu, Justin Johnson, Li Fei-Fei, Jiajun Wu, [0.5em] Stanford University, University of Michigan · 2025

Read Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization by Kyle Sargent, Kyle Hsu, Justin Johnson, Li Fei-Fei, Jiajun Wu, [0.5em] Stanford University, University of Michigan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11056.txt.

Introduction

Kyle Sargent, Kyle Hsu, Justin Johnson, Li Fei-Fei, and Jiajun Wu present FlowMo in two thousand twenty-five. FlowMo is a diffusion autoencoder that tokenizes images for generation. Two stage systems first compress images, then learn to generate. Past diffusion tokenizers lagged top scores on ImageNet rebuilds.

Their training splits into mode matching and mode seeking stages. That split is the key insight behind the design. No convolutions, adversarial losses, or distilled teachers are used. Latent codes need not align to a spatial grid either. Scores set new best marks at several compression rates.

Catalog record

CREATORKyle Sargent, Kyle Hsu, Justin Johnson, Li Fei-Fei, Jiajun Wu, [0.5em] Stanford University, University of Michigan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11056

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization Kyle Sargent, Kyle Hsu, Justin Johnson, Li Fei-Fei, Jiajun Wu, [0.5em] Stanford University, University of Michigan Abstract Since the advent of popular visual generation frameworks like VQGAN and latent diffusion models, state-of-the-art image generation systems have generally been two-stage systems that first tokenize or compress visual data into a lower-dimensional latent space before learning a generative model. Tokenizer training typically follows a standard recipe in which images are compressed and reconstructed subject to a combination of MSE, perceptual, and adversarial losses. Diffusion autoencoders have been proposed in prior work as a way to learn end-to-end perceptually-oriented image compression, but have not yet shown state-of-the-art performance on the competitive task of ImageNet-1K reconstruction. We propose FlowMo, a transformer-based diffusion autoencoder that achieves a new state-of-the-art for image tokenization at multiple compression rates without using convolutions, adversarial losses, spatially-aligned two-dimensional latent codes, or distilling from other tokenizers. Our key insight is that FlowMo training should be broken into a mode-matching pre-training stage and a mode-seeking post-training stage. In addition, we conduct extensive analyses and explore the training of generative models atop the FlowMo tokenizer. Our code and models are available at https://kylesargent.github.io/flowmo . 1 Introduction Figure 1 : Discrete tokenizer comparison. State-of-the-art discrete tokenizers are benchmarked by encoding and reconstructing the ImageNet-1K validation dataset at 256\times 256 resolution. with performance measured in reconstruction FID (rFID), which trades off against compression rate as measured in bits per pixel (BPP). Whether trained for reconstruction at a low BPP (FlowMo-Lo) or high BPP (FlowMo-Hi), FlowMo achieves state-of-the art image tokenization performance compared with the respective baselines. Moreover, FlowMo is a transformer-based diffusion autoencoder which does not use convolutions, adversarial losses, or proxy objectives from auxiliary tokenizers. Generative models such as diffusion models [ 50 , 51 , 24 , 17 , 37 , 36 ] and discrete autoregressive models [ 16 , 8 ] have seen compelling applications in the creation of image and video content [ 6 , 48 , 32 ] . Although less common systems such as cascaded [ 48 ] and single-stage [ 21 , 25 ] systems have been explored, state-of-the-art systems for visual generation generally consist of two stages: first, a “tokenizer” is learned to compress pixel data to a smaller and more tractable discrete or continuous latent space; second, a generative model is trained over this compressed latent space. Due to the dominance of this two-stage paradigm, tokenization has become an active research area in its own right, with multiple influential works focusing heavily on tokenizer training and design [ 63 , 38 , 57 , 62 , 18 ] . In this work, we focus on tokenizers with a discrete latent space. Original image OpenMagViT-V2 FlowMo-Lo (Ours) LlamaGen-32 FlowMo-Hi (Ours) (rFID=1.17) (rFID=0.95) (rFID=0.59) (rFID=0.56) Figure 2 : Example reconstructions. Comparison of original and reconstructed images of faces and text. OpenMagViT-V2 and FlowMo-Lo are 0.07-bits per pixel tokenizers to be compared against each other. LlamaGen-32 and FlowMo-Hi are 0.22-bits per pixel tokenizers to be compared against each other. Best viewed zoomed-in in the electronic version. More comparisons are available on our website . Since VQGAN [ 16 ] , a dominant architecture and training scheme for discrete image tokenizers has emerged. State-of-the-art image tokenizers are typically convolutional autoencoders trained to downsample visual data to a 2-dimensional, spatially-aligned latent code, and then upsample the latent code to a reconstruction regularized by adversarial and perceptual losses. Various deviations from this setup have been proposed: TiTok [ 63 ] used a transformer-based architecture and 1-dimensional latent code although relying on a traditional CNN-based tokenizer for an initial distillation stage, and ViT-VQGAN [ 61 ] used a transformer-based encoder and decoder. In general, however, the same basic setup for image tokenization has remained dominant. In this work, we propose FlowMo ( Flow to the Mo de), a significant technical departure from the current state-of-the-art for image tokenization. First, in tackling the task of image tokenization, we propose to model the multimodal reconstruction distribution using a rectified flow objective for the decoder [ 37 , 36 ] . Second, we use a fully transformer-based architecture to encode to and decode from a 1-dimensional latent code. Third, we optimize FlowMo end-to-end, without distilling from a pre-existing 2-dimensional tokenizer [ 63 ] or encoding on top of a latent space from a 2-dimensional tokenizer [ 1 ] . Most importantly, FlowMo achieves state-of-the-art performance via a key insight. FlowMo is a diffusion autoencoder, a system which has been explored in prior work [ 46 , 2 , 26 , 60 ] . Yet, the state-of-the-art for the most competitive perceptual reconstruction benchmark, ImageNet-1K [ 13 ] , continues to be dominated by more traditional CNN- or GAN-based tokenizers. FlowMo achieves state-of-the-art tokenization via the following key insight: for the task of perceptual reconstruction, it is better to sample modes of the reconstruction distribution that are perceptually close to the original image than to try to match all modes. Hence, we divide FlowMo training into a mode-matching pre-training stage , in which the system is trained end-to-end with a diffusion loss on the decoder like typical diffusion autoencoders, and a mode-seeking post-training stage , in which the system is trained to selectively drop modes of the reconstruction distribution which are not close to the original image. We explain both stages in Section 3 . Despite its significant methodological departures from prior work, FlowMo is state-of-the-art when compared with the strongest available tokenizers at multiple BPPs. Our main contributions are as follows: • We present a simple but novel architecture for image tokenization based on diffusion autoencoders [ 46 ] and the multimodal diffusion image transformer (MMDiT) [ 17 ] . • We present a novel two-stage training scheme for diffusion autoencoders, consisting of mode-matching pre-training and mode-seeking post-training . • We set a new state-of-the-art for perceptual image tokenization in the 0.07 bits per pixel and 0.22 bits per pixel regimes, in terms of rFID, PSNR, and other metrics. We also show that a generative model trained with a FlowMo tokenizer can match (though not beat) the performance of a generative model trained with a traditional tokenizer. • We conduct an extensive analysis of the system design choices in FlowMo, outlining several subtle but critical decisions in the noise schedule, sampler design, model design, quantization strategy, and post-training. 2 Related Work Image tokenization. State-of-the-art systems for visual generation generally consist of two stages (with a few notable exceptions attempting to learn directly in pixel space [ 25 , 27 , 21 , 12 ] ). The first stage is a “tokenizer” which reduces the spatial or spatiotemporal dimension of pixel data by projecting to a continuous or discrete latent space. The dominance of this paradigm has led to numerous works that study tokenizers as important components in their own right [ 9 , 63 , 57 , 62 , 38 , 53 ] . In this work, we study tokenizers with discrete latents. Unlike prior work, FlowMo is the first diffusion autoencoder-based architecture to achieve state-of-the-art performance on ImageNet-1K reconstruction. Diffusion autoencoders. Diffusion models are popular for visual generation [ 47 , 6 ] ,

Read the complete text