MIMFlow: Integrating Masked Image Modeling with Normalizing Flows for End-to-End Image Generation
Yang Chen, Xiaowei Xu, Shuai Wang, Xinwen Zhang, Qiushi Guo, Tiezheng Ge, Limin Wang · 2026
Read MIMFlow: Integrating Masked Image Modeling with Normalizing Flows for End-to-End Image Generation by Yang Chen, Xiaowei Xu, Shuai Wang, Xinwen Zhang, Qiushi Guo, Tiezheng Ge, Limin Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26016.txt.
Introduction
Yang Chen and colleagues fuse two image-generation ideas that usually sit apart. Normalizing flows can state exact probabilities for images and sample from them. Their strict invertibility wastes capacity on pixel-level noise. High-level structure then suffers. Masked image modelling, by contrast, learns rich semantic features. It rarely joins a full generation pipeline.
MIMFlow joins them end to end. A variational encoder reads masked images and infers a semantic latent code. The flow models only that simpler, low-frequency semantic space. A specialist decoder then adds the high-frequency detail. On ImageNet at 256-pixel size, the large variant reaches strong linear probing accuracy. Its generation quality score beats similar-scale flow baselines by a wide margin. It does so with half the usual token count. Global structure stays coherent because the flow no longer fights pixel noise.
Catalog record
| CREATOR | Yang Chen, Xiaowei Xu, Shuai Wang, Xinwen Zhang, Qiushi Guo, Tiezheng Ge, Limin Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26016
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MIMFlow: Integrating Masked Image Modeling with Normalizing Flows for End-to-End Image Generation Yang Chen, Xiaowei Xu, Shuai Wang, Xinwen Zhang, Qiushi Guo, Tiezheng Ge, Limin Wang Abstract Normalizing Flows (NFs) are powerful generative models capable of exact density estimation and sampling. However, their strict invertibility often forces the model to exhaust its capacity on low-level pixel details, hindering the capture of high-level semantic structures. While Masked Image Modeling (MIM) has excelled in representation learning, its integration into generative pipelines has remained largely modular and disjointed. In this paper, we propose MIMFlow, a unified end-to-end framework that jointly optimizes latent semantics, pixel reconstruction, and generative flow. By employing a VAE encoder to infer semantic latent from masked images, MIMFlow achieves a principled decoupling of the generative task: the Normalizing Flow focuses on modeling a simplified, low-frequency semantic manifold, while a specialized decoder handles high-frequency synthesis. This design effectively resolves the inherent capacity bottleneck of NFs, allowing the model to prioritize global structural coherence over redundant noise. Empirical results on ImageNet 256 × \times 256 show that MIMFlow-L reaches 71.3% linear probing accuracy and an FID of 2.50. Despite using only 128 tokens (50% fewer than standard models), it yields a 32.8% performance gain over similar-scale NF baselines. 1 Introduction Normalizing Flows (NFs) are generative models that map a complex data distribution to a simple prior distribution through a series of invertible and differentiable transformations [ 8 , 23 , 52 , 4 , 53 , 55 , 42 , 43 , 15 , 39 , 16 , 27 ] . A primary advantage of NFs is their ability to perform both exact density estimation and sampling within a single network. Besides, from the perspective of flow matching, NFs can be viewed as the expansion of an Ordinary Differential Equation (ODE), allowing for end-to-end optimization of the entire flow process [ 2 ] . Recent work, such as SimFlow [ 53 ] , has further demonstrated that NFs can provide highly expressive priors for Variational Autoencoders (VAEs) by leveraging their likelihood estimation capabilities. However, strict invertibility forces NFs to prioritize low-level pixel details over high-level semantics, exhausting model capacity and hindering generative quality. Figure 1 : MIM in Different Paradigms. (a) Self-Supervised Learning: Employs high-ratio masking as a self-supervised proxy task for representation learning. (b) Generative Tokenizers: A two-stage approach where the latent space is pre-trained with MIM before training a separate generative model. (c) MIMFlow (Ours): A unified framework that jointly optimizes latent semantics, pixel reconstruction, and generative flow in an end-to-end manner. While NFs struggle to capture global structures, Masked Image Modeling (MIM, Fig. 1 a) has established itself as a cornerstone of self-supervised representation learning [ 18 , 46 ] . Despite its success in discriminative tasks, the role of MIM in generative modeling remains relatively under-explored. Recent literature suggests that integrating discriminative features can significantly bolster generative performance, with some approaches distilling knowledge from pre-trained representation models to guide the generation process [ 51 , 26 , 57 ] . Furthermore, RAE [ 54 , 40 ] directly employs representation models as VAE encoders. Specifically, they found that MAE lags behind DINO in these generative frameworks. This likely stems from the mismatch between MAE’s high-mask reconstruction and the continuous distribution modeling required for synthesis. Thus, whether semantic-focused MIM can effectively enhance generative models remains an open question. Recent attempts to incorporate MIM into generative pipelines have primarily focused on refining visual tokenizers. Specifically, works such as MAETok [ 1 ] and DeTok [ 48 ] have pioneered the use of masking as a denoising or discriminative objective to enhance the robustness of latent structures. Building on this, VTP [ 49 ] demonstrated that unifying MIM with contrastive learning can further foster a high-level understanding essential for tokenizer scalability. However, these methods largely treat the tokenizer as a modular, standalone precursor, separating representation learning from the core generative process ( Fig. 1 b). In contrast, our method jointly optimizes representation, reconstruction and generation in an end-to-end framework ( Fig. 1 c). This design enables a principled decoupling of the generative task: the NF is dedicated to modeling the low-frequency semantic manifold, while a specialized decoder handles high-frequency synthesis. By reducing the burden on NFs to capture pixel-level noise, MIMFlow mitigates a central capacity bottleneck of traditional NFs and offers an NF-oriented recipe for more semantic latent modeling. Specifically, our MIMFlow employs a VAE encoder with learnable tokens to extract stable latent representations from masked images. These latents are then optimized through a dual-objective framework: a NF performs exact density estimation, while a decoder focuses on pixel-level reconstruction. This architecture facilitates a strategic division of labor: by mandating the encoder to infer missing spatial context, the resulting latent space is biased toward global structural coherence rather than redundant local noise. Consequently, the NF can model a simpler semantic manifold instead of dense pixel-level correlations. Crucially, linear probing evaluations reveal a substantial increase in classification accuracy within the latent space, validating the enhanced semantic quality of our representations. Finally, on the ImageNet 256×256 benchmark, MIMFlow improves similar-scale NF baselines while using fewer latent tokens. In summary, our main contributions are as follows: • MIMFlow Framework: We propose an end-to-end framework that unifies representation and generation, moving beyond the modular limitations of existing tokenizer-based methods. • Principled Decoupling: We introduce a strategy to decouple semantic modeling from pixel-level synthesis, effectively resolving the inherent capacity bottleneck of NFs. • Empirical Excellence: We demonstrate that MIMFlow significantly enhances latent semantics and achieves superior generative performance on the ImageNet 256\times 256 benchmark. 2 Related Work Normalizing Flows [ 36 , 31 , 24 , 7 , 8 , 23 , 9 , 13 , 10 , 17 , 21 ] provide a mathematically principled framework for bidirectional mapping between data and latent spaces. Early works like RealNVP [ 8 ] and Glow [ 23 ] established exact log-likelihood estimation through coupling layers, though they struggled to scale to high-resolution synthesis. Recent advancements [ 42 , 14 , 52 , 4 ] have revitalized the field by integrating autoregressive Transformers and latent-space modeling, effectively leveraging NFs for both high-fidelity generation and semantic alignment. Building on this, SimFlow [ 53 ] demonstrates that NFs can serve as highly expressive probability estimators to replace restrictive VAE priors, significantly enhancing generative performance. While these models focus on architectural scaling or post-hoc alignment, our work is the first to integrate MIM objectives directly into the end-to-end training of NFs. By unifying self-supervised representation learning with generative flow optimization, we fully exploit the dual potential of NFs for simultaneous robust feature extraction and high-quality synthesis. Masked Image Modeling has emerged as a dominant paradigm in self-supervised learning, with methods like MAE [ 18 ] and SimMIM [ 46 ] demonstrating that reconstructing masked inputs forces encoders to learn robust structural features. While initially designed for discriminative tasks, it has also been utilized in generative frameworks. Specifically, recent literature explores distilling knowledge from pre-trained vision foundation models to guide diffusion processes [ 51 , 57 ] . Within the context of visual tokenizers, MAETok [ 1 ] and DeTok [ 48 ] employ masking as a denoising or discriminative auxiliary task to