Effective and Efficient Masked Image Generation Models
Zebin You, Jingyang Ou, Xiaolu Zhang, Jun Hu, Jun Zhou, Chongxuan Li · 2025
Read Effective and Efficient Masked Image Generation Models by Zebin You, Jingyang Ou, Xiaolu Zhang, Jun Hu, Jun Zhou, Chongxuan Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07197.txt.
Introduction
Zebin You and co-authors unify two families of image generators into one efficient design in a 2025 paper. Masked generation models build pictures by unmasking patches step by step, while masked diffusion models arrived from a different tradition. The authors show the two are really one framework wearing different clothes, then explore its training and sampling knobs systematically.
Their tuned model, eMIGM, generates sharp images on the standard ImageNet benchmark. At matched size and sampling cost it beats a well-known rival, and when scaled up it rivals the best continuous diffusion models while using under half the sampling steps. At the largest test size it passes another strong diffusion baseline outright. Faster generation at equal quality cuts the cost of image synthesis for everyone downstream.
Catalog record
| CREATOR | Zebin You, Jingyang Ou, Xiaolu Zhang, Jun Hu, Jun Zhou, Chongxuan Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07197
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Effective and Efficient Masked Image Generation Models Zebin You, Jingyang Ou, Xiaolu Zhang, Jun Hu, Jun Zhou, Chongxuan Li Abstract Although masked image generation models and masked diffusion models are designed with different motivations and objectives, we observe that they can be unified within a single framework. Building upon this insight, we carefully explore the design space of training and sampling, identifying key factors that contribute to both performance and efficiency. Based on the improvements observed during this exploration, we develop our model, referred to as eMIGM . Empirically, eMIGM demonstrates strong performance on ImageNet generation, as measured by Fréchet Inception Distance (FID). In particular, on ImageNet 256 × 256 256\times 256 , with similar number of function evaluations (NFEs) and model parameters, eMIGM outperforms the seminal VAR. Moreover, as NFE and model parameters increase, eMIGM achieves performance comparable to the state-of-the-art continuous diffusion models while requiring less than 40% of the NFE. Additionally, on ImageNet 512 × 512 512\times 512 , with only about 60% of the NFE, eMIGM outperforms the state-of-the-art continuous diffusion models. Code is available at https://github.com/ML-GSAI/eMIGM . † † affiliationnotice: \icmlItern \icmlProjectLeaders 1 Introduction Masked modeling has proven effective across various domains, including self-supervised learning [ 18 , 4 , 12 ] , image generation [ 29 , 8 , 30 ] , and text generation [ 41 , 43 , 31 ] . In image generation, MaskGIT [ 8 ] introduced masked image generation, offering efficiency and quality improvements over autoregressive models but still lagging behind diffusion models [ 22 , 44 , 45 ] due to information loss from discrete tokenization [ 14 , 49 ] . MAR [ 30 ] eliminated this bottleneck via diffusion loss, achieving strong results, yet key factors (e.g., masking schedule, loss function) remain underexplored. Moreover, with limited sampling steps (e.g., 16), its performance falls short of coarse-to-fine next-scale prediction model VAR [ 48 ] . In parallel, masked diffusion models [ 41 , 43 , 31 , 38 ] have shown promise in text generation, demonstrating scaling properties [ 37 ] similar to ARMs and offering a principled probabilistic framework for training and inference. However, their applicability to image generation remains an open question. Figure 1: Generated samples from eMIGM trained on ImageNet 512\times 512 . We propose a unified framework integrating masked image modeling [ 8 , 30 ] and masked diffusion models [ 31 , 41 , 43 ] , leveraging the strengths of both paradigms. This enables a systematic exploration of training and sampling strategies to optimize performance. For training, we find that images, due to their high redundancy, benefit from a higher masking ratio, a simple weighting function inspired by MaskGIT and MAE [ 18 ] tricks, improving generation quality. We also introduce CFG with Mask, replacing the fake class token with a mask token for unconditional generation, further enhancing performance. For sampling, predicting fewer tokens in early stages improves results. However, early-stage guidance decreases variance, raising FID. To counter this, we propose a time interval strategy for classifier-free guidance, applying guidance only in later stages. This maintains strong performance while significantly accelerating sampling by reducing NFEs. Building on our training and sampling improvements, we develop eMIGM and evaluate it on ImageNet [ 11 ] at 256\times 256 and 512\times 512 resolutions. As model parameters scale, eMIGM achieves progressively higher sample quality in a predictable manner (Fig. 4(a) ). Larger models further enhance efficiency, maintaining superior quality with similar training FLOPs and sampling time (Fig. 4(b) , Fig. 4(c) ). Notably, eMIGM delivers high-quality samples with few sampling steps. On ImageNet 256\times 256 , with similar NFEs and model parameters, it consistently outperforms VAR [ 48 ] . Increasing NFE and model size, our best-performing eMIGM-H becomes comparable to state-of-the-art diffusion models like REPA [ 51 ] (FID 1.57 vs. 1.42)—without requiring self-supervised features. On ImageNet 512\times 512 , eMIGM-L surpasses EDM2 [ 26 ] while using only 60% of its NFEs, demonstrating efficiency and scalability. Qualitatively, eMIGM generates realistic and diverse images (Fig. 1 ). In summary, our key contributions are as follows: • We propose a unified formulation to systematically explore the design space of masked image generation models, uncovering the role of each component. • We introduce the time interval strategy for classifier-free guidance, maintaining high performance while significantly reducing sampling time. • We surpass the state-of-the-art diffusion models on ImageNet 512\times 512 with only 60% of NFEs. • We demonstrate that eMIGM benefits from scaling, with larger eMIGM models achieving greater efficiency. 2 Preliminaries 2.1 Masked Image Generation Let \boldsymbol{x}=[\boldsymbol{x}^{i}]_{i=1}^{N} represent the discrete tokens of an image obtained via a VQ encoder [ 14 , 49 ] , and let [M] denote the special mask token. We consider two seminal masked image generation methods. MaskGIT [ 8 ] first extends the concept of masked language modeling from BERT [ 12 ] (i.e., predicting masked tokens based on unmasked tokens) to image generation, achieving excellent performance with low sampling cost (approximately 10 sampling steps) on ImageNet [ 11 ] . However, its performance degrades when the number of sampling steps increases under its default mask schedule. During training, MaskGIT optimizes the cross entropy loss as follows. A ratio r is sampled from [0,1] , and based on the mask scheduling function \gamma_{r} , masked image \boldsymbol{x}_{\overline{\textbf{M}}} is sampled from masking distribution q_{\gamma_{r}}(\boldsymbol{x}_{\overline{\textbf{M}}}|\boldsymbol{x}) that randomly masks \lceil N\gamma_{r}\rceil tokens of \boldsymbol{x} as [M] . The loss function is then defined as: \mathcal{L}(\boldsymbol{x})=\mathbb{E}_{r\sim U[0,1]}\mathbb{E}_{q_{\gamma_{r}}(\boldsymbol{x}_{\overline{\textbf{M}}}|\boldsymbol{x})}\left[\sum_{\{i|\boldsymbol{x}^{i}=\text{[M]}\}}\!\!\!\!\!\!-\log p_{\boldsymbol{\theta}}\left(\boldsymbol{x}^{i}\,|\,\boldsymbol{x}_{\overline{\textbf{M}}}\right)\right]. (1) Table 1: Comparison of different masked image modeling approaches through a unified framework. The differences among these approaches are defined by the choice of masking distribution q(\boldsymbol{x}_{t}|\boldsymbol{x}_{0}) , weighting function w(t) , and conditional distribution p_{\boldsymbol{\theta}}(\boldsymbol{x}_{0}^{i}\mid\boldsymbol{x}_{t}) . Method Masking Distribution Weighting Function Conditional Distribution q(\boldsymbol{x}_{t}|\boldsymbol{x}_{0}) w(t) p_{\boldsymbol{\theta}}(\boldsymbol{x}_{0}^{i}\mid\boldsymbol{x}_{t}) MaskGIT Uniformly mask \lceil N\gamma_{t}\rceil tokens w/o replacement w(t)=1 Categorical Distribution MAR Uniformly mask \lceil N\gamma_{t}\rceil tokens w/o replacement w(t)=1 Diffusion Model MDM Mask N tokens independently with ratio \gamma_{t} w(t)=\frac{\gamma_{t}^{\prime}}{\gamma_{t}} Categorical Distribution During sampling, MaskGIT starts with an image where all tokens are masked, \boldsymbol{x}_{0} . For each iteration t\in\{1,2,\dots,T\} , the number of masked tokens is n_{t}=\lceil\gamma_{\frac{t}{T}}N\rceil , and the model receives input \boldsymbol{x}_{\frac{t-1}{T}} . The model predicts the probabilities for all tokens, and the \hat{n}_{t}=n_{t-1}-n_{t} tokens with the highest confidence are unmasked, updating to \boldsymbol{x}_{\frac{t}{T}} . MAR [ 30 ] proposes using a diffusion model [ 44 ] to model the per-token distribution, which eliminates the need for discrete tokenizers. By avoiding the information loss of discrete tokenizers, MAR achieves excellent image generation performance. During training, MAR samples the masking ratio m_{r} from a truncated Gaussian distribution with mean 1.0, standard deviation 0.25, truncated to [0.7, 1.0]. For sampling, MAR adopts a decoding strategy similar to that of MaskGIT. 2.2 Masked Diffusion Models Let \boldsymbol{x}=[\boldsymbol{x}^{i}]_{i=1}^{N} represent the discrete text tokens of a sentence, [M] denote the special mask token, and \gamma_{t} represent the mask schedule. MDMs [ 32 , 43 , 41 ] gradually add masks to the data in the forward process and remove them during the reverse process. Here, we focus on the parameterized form of RADD [ 38 ] . Given a noise level t\in[0,1] , the forward process of MDM is defined