MeanFlow Transformers with Representation Autoencoders
Zheyuan Hu, Chieh-Hsin Lai, Yuki Mitsufuji, Stefano Ermon · 2025
Read MeanFlow Transformers with Representation Autoencoders by Zheyuan Hu, Chieh-Hsin Lai, Yuki Mitsufuji, Stefano Ermon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13019.txt.
Introduction
Zheyuan Hu and four co-authors make one-step image generation cheaper and steadier. MeanFlow models learn to jump directly from noise to an image in a few steps. Training them in the usual autoencoder's latent space is expensive and unstable, and the decoder dominates sampling cost. The team moves MeanFlow into the latent space of a representation autoencoder, whose latents come from a pretrained vision encoder and carry rich semantics, paired with a lightweight decoder. Naive training there explodes gradients, so they stabilize it with trajectory-aware initialization and a two-stage scheme distilling from a flow matching teacher, then bootstrapping.
The payoff on ImageNet is concrete: one-step generation quality improved markedly over vanilla MeanFlow, sampling cost fell by over a third. Total training cost dropped by more than four fifths, with competitive results at the larger resolution too, and no guidance hyperparameters to tune.
Catalog record
| CREATOR | Zheyuan Hu, Chieh-Hsin Lai, Yuki Mitsufuji, Stefano Ermon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13019
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MeanFlow Transformers with Representation Autoencoders Zheyuan Hu, Chieh-Hsin Lai, Yuki Mitsufuji, Stefano Ermon Abstract MeanFlow (MF) is a diffusion-motivated generative model that enables efficient few-step generation by learning long jumps directly from noise to data. In practice, it is often used as a latent MF by leveraging the pre-trained Stable Diffusion variational autoencoder (SD-VAE) for high-dimensional data modeling. However, MF training remains computationally demanding and is often unstable. During inference, the SD-VAE decoder dominates the generation cost, and MF depends on complex guidance hyperparameters for class-conditional generation. In this work, we develop an efficient training and sampling scheme for MF in the latent space of a Representation Autoencoder (RAE), where a pre-trained vision encoder (e.g., DINO) provides semantically rich latents paired with a lightweight decoder. We observe that naive MF training in the RAE latent space suffers from severe gradient explosion. To stabilize and accelerate training, we adopt Consistency Mid-Training for trajectory-aware initialization and use a two-stage scheme: distillation from a pre-trained flow matching teacher to speed convergence and reduce variance, followed by an optional bootstrapping stage with a one-point velocity estimator to further reduce deviation from the oracle mean flow. This design removes the need for guidance, simplifies training configurations, and reduces computation in both training and sampling. Empirically, our method achieves a 1-step FID of 2.03, outperforming vanilla MF’s 3.43, while reducing sampling GFLOPS by 38% and total training cost by 83% on ImageNet 256. We further scale our approach to ImageNet 512, achieving a competitive one-step FID of 3.23 with the lowest GFLOPS among all baselines. Code is available at https://github.com/sony/mf-rae . Figure 1 : Overview of our method’s advantages. On ImageNet 256, vanilla MF (a) employs the slow SD-VAE decoder, which accounts for 73% of the total generation cost and thus bottlenecks the few-step generation speed. In contrast, our MF-RAE (b) leverages a higher-dimensional RAE latent space with semantically rich features and an efficient decoder. The DiT {}^{\text{DH}} architecture is adopted to effectively process the high-dimensional latent space. As a result, while the converged 1-step FID of vanilla MF is 3.43 after more than 600 H100 GPU-days, our MF-RAE achieves a superior FID of 2.03 in only 100 H100 GPU-days. Additionally, our total generation cost is reduced by 38% compared to vanilla MF in terms of GFLOPS, even with the same 1-step generation setting. 1 Introduction Diffusion models (or flow matching) ( 32 ; 36 ; 12 ; 37 ) have been shown to achieve high-fidelity sample generation. Its sampling can be interpreted as solving the associated probability flow ordinary differential equation (PF-ODE) ( 37 ) . However, this procedure requires many neural network evaluations to approximate the numerical integration, which makes diffusion model generation notoriously slow. Recent research ( 34 ; 17 ; 9 ; 20 ) has shifted toward flow map models , with a promising representative called MeanFlow (MF) ( 9 ) . These models directly learn the PF-ODE solution map, transporting any initial state at one time to the corresponding state on the same trajectory at another time. As a result, they enable generation in a few steps, mapping pure noise to clean data with only a small number of network evaluations. In practice, MF is often used in a latent space by leveraging a pre-trained Stable Diffusion variational autoencoder (SD-VAE) ( 28 ) for high-dimensional image generation. Despite these advances in diffusion-based few-step generative models, MF training and inference remain inefficient for such high-dimensional latent representations. Training MF, even in a latent space, still requires hundreds of H100 GPU-days ( 49 ) on high-dimensional datasets such as ImageNet 256 ( 5 ) . It is further complicated by intricate classifier-free guidance (CFG) ( 13 ) configurations for class-conditional generation, which involve two CFG scale hyperparameters and two additional hyperparameters controlling the CFG triggering interval. These hyperparameters must be carefully tuned through extensive grid search to maximize MF performance, thereby increasing the overall complexity of training. Moreover, the Jacobian vector product (JVP) required by the MF loss introduces an additional source of computational cost and instability. Even when it is computed using the most efficient forward mode automatic differentiation ( 31 ) , the JVP remains a significant bottleneck during training. Supporting JVP in modern components such as Flash Attention ( 4 ) also requires extra implementation effort ( 47 ) , making MF cumbersome and time-consuming to adapt to new model architectures. Regarding the inference, although MF enables few-step generation, the computational cost of the SD-VAE decoder that maps generated latent vectors back to pixel space dominates and substantially slows down the overall generation speed ( 18 ) . Specifically, as shown in Figure 1 (a), for vanilla MF with a conventional DiT/SiT architecture combined with SD-VAE on ImageNet 256, the DiT/SiT requires 114 GFLOPS, whereas the SD-VAE decoder consumes 310 GFLOPS, which indicates that approximately 73% of the total computation is spent on the decoder. Recent research on the Representation Autoencoder (RAE) ( 46 ) replaces the conventional SD-VAE in latent diffusion models with a frozen pre-trained representation encoder (e.g., DINO ( 2 ) ) and trains only a ViT-based ( 7 ) decoder on top of its latent tokens. Unlike the classic SD-VAE that uses a U-Net backbone to compress images into a low-dimensional latent space, RAE modernizes this design by adopting a transformer architecture and using semantically rich, high-dimensional representations as the generative space. To model these higher-dimensional latents, RAE augments a DiT backbone with a wide yet lightweight DDT head ( 40 ) tailored for latent diffusion modeling, yielding an expressive and efficient DiT {}^{\text{DH}} architecture. For latent diffusion models, RAE brings limited improvement in sampling speed, since the main bottleneck is the large number of network evaluations required to solve the latent PF-ODE. However, we emphasize that RAE’s decoder efficiency is particularly crucial for few-step models such as MF: its decoder requires about 106 GFLOPS, nearly a 3\times reduction compared to the \sim 310 GFLOPS SD-VAE decoder used in vanilla MF. This directly alleviates the decoder side bottleneck in current few-step generation and is a key motivation for our approach. To this end, we improve the efficiency of MF training and sampling by learning MF in the RAE latent space, and show that our approach is stable, fast, and high-quality in practice. We systematically analyze and decompose MF training into the following components. First, we observe that naively training MF with the DiT {}^{\text{DH}} architecture in the RAE latent space is unstable: gradients explode early in training, regardless of whether MF is initialized randomly or from a pre-trained flow matching teacher. We attribute this to a mismatch between the training signal of flow matching and the objective of MF: flow matching learns infinitesimal transitions along the PF-ODE trajectory, whereas MF must learn long jumps between distant time steps, and random initialization further aggravates this issue. To address this, we initialize MF with weights obtained from Consistency Mid-Training (CMT) ( 14 ) , which learns a trajectory-aware initialization by following the numerical PF-ODE trajectory of a pre-trained flow matching model. Second, after stabilizing MF training with CMT, we adopt the MeanFlow Distillation (MFD) algorithm, which efficiently converts a pre-trained flow matching model (teacher) into a few-step MF