D-AR: Diffusion via Autoregressive Models
Ziteng Gao, Mike Zheng ShouShow Lab, National University of Singapore · 2025
Read D-AR: Diffusion via Autoregressive Models by Ziteng Gao, Mike Zheng ShouShow Lab, National University of Singapore (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23660.txt.
Introduction
Ziteng Gao and Mike Zheng Shou present D-AR, a way to make images by recasting diffusion as plain next-token prediction, in this 2025 arXiv paper. A tokenizer first turns a picture into a string of symbols. Early symbols decode into the rough first pass of a cleanup process, and later ones into finer passes. That coarse-to-fine order matches how autoregressive models naturally write.
A standard language-model backbone then generates the symbols one by one. Nothing about its masks or training routines changes. Each new chunk can be decoded into the matching cleanup step on the fly. Partial generations give consistent previews, and layout control works without retraining. On ImageNet the system scored 2.09 FID with a 775-million-parameter backbone.
Catalog record
| CREATOR | Ziteng Gao, Mike Zheng ShouShow Lab, National University of Singapore |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23660
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
D-AR: Diffusion via Autoregressive Models Ziteng Gao, Mike Zheng ShouShow Lab, National University of Singapore Abstract This paper presents Diffusion via Autoregressive models (D-AR), a new paradigm recasting the image diffusion process as a vanilla autoregressive procedure in the standard next-token-prediction fashion. We start by designing the tokenizer that converts images into sequences of discrete tokens, where tokens in different positions can be decoded into different diffusion denoising steps in the pixel space. Thanks to the diffusion properties, these tokens naturally follow a coarse-to-fine order, which directly lends itself to autoregressive modeling. Therefore, we apply standard next-token prediction on these tokens, without modifying any underlying designs (either causal masks or training/inference strategies), and such sequential autoregressive token generation directly mirrors the diffusion procedure in image space. That is, once the autoregressive model generates an increment of tokens, we can directly decode these tokens into the corresponding diffusion denoising step in the streaming manner. Our pipeline naturally reveals several intriguing properties, for example, it supports consistent previews when generating only a subset of tokens and enables zero-shot layout-controlled synthesis. On the standard ImageNet benchmark, our method achieves 2.09 FID using a 775M Llama backbone with 256 discrete tokens. We hope our work can inspire future research on unified autoregressive architectures of visual synthesis, especially with large language models. Code and models will be available at https://github.com/showlab/D-AR . Figure 1: Diffusion via autoregressive modeling (D-AR) framework for visual generation. As the autoregressive transformer generates tokens, D-AR can simultaneously perform corresponding diffusion steps via token conditioning and jump-estimate target samples as rough previews effortlessly. † † {\dagger} Corresponding author. Figure 2: Uncurated generated samples from D-AR-XL with 256\times 256 resolutions (CFG=4.0). 1 Introduction Autoregressive models, exemplified by large language models (LLMs) [ 1 , 2 , 3 ] , have emerged as the foundation of modern NLP, achieving state-of-the-art performance with the simple next token prediction paradigm. With their widespread adoption, the autoregressive next token prediction paradigm has established itself as the de facto standard in modern AI and this has fostered software ecosystem for optimizing such training and inference pipelines [ 4 , 5 , 6 ] . The remarkable success of autoregressive models in language has also inspired exploration into visual generation tasks [ 7 , 8 , 9 ] , with the broader goal of building unified frameworks of both vision and language [ 10 , 11 , 12 , 13 ] . However, unlike text, where sequential structure is naturally defined, images lack an inherently linear ordering, posing challenges for adapting such paradigm to vision modeling. Recent studies explore different visual orderings in autoregressive modeling [ 14 , 15 , 16 , 17 , 18 ] . However, these approaches typically require significant modifications to the core mechanisms, often deviating from the standard next token prediction objective. In parallel, current vision generation pipelines are most dominated by diffusion models [ 19 , 20 , 21 ] , which underpin several commercial systems [ 22 , 23 , 24 ] . Diffusion models are particularly excelling at modeling continuous signals: starting from random noise, they iteratively refine input through denoising to produce high-quality images. However, despite impressive visual fidelity, the diffusion sampling process, operating in a dense manner, is inherently slow, especially with models with large parameters and significant sampling steps. Furthermore, the diffusion architecture poses challenges for seamless integration with LLMs and limit their potential in unified multimodal systems. In this paper, we aim to bridge the diffusion process and autoregressive modeling for visual generation, leveraging strengths from both paradigms. Crucially, we maintain a strict adherence to the standard next-token prediction paradigm, without any modifications to the underlying autoregressive mechanism. To achieve this, we present the coarse-to-fine sequential diffusion tokenizer to reinterpret the diffusion process on raw image pixels as a sequence of discrete tokens. In this formulation, early tokens represent conditions in early diffusion steps from pure noise, whilst later tokens capture progressive steps over less noised inputs, leading to a naturally linearized decomposition of visual sequence [ 25 ] . We design the diffusion model in the proposed diffusion tokenizer to be light and fast, e.g., with around 185M parameters and 8 diffusion steps, and achieve 1.52 rFID on ImageNet [ 26 ] with a total budget of 256 discrete tokens. With this design, we can perform the diffusion process on image pixels via predicting next token in token sequence with the autoregressive mechanism untouched. Therefore, we name this framework as D ffusion via A uto r egressive models ( D-AR ). D-AR offers several intriguing properties inherited from both diffusion and autoregressive worlds, including 1) it natively supports fast inference with KV cache, 2) it provides consistent previews when partial tokens are generated, and 3) it can be easily extended to zero-shot layout-controlled synthesis by conditioning several prefix tokens. Besides these interesting properties, D-AR also excels on the standard ImageNet class-conditioned generation benchmark. With a 775M-parameter, LLaMA backbone [ 2 ] operating purely in the next-token-prediction paradigm, D-AR-XL achieves 2.09 gFID with a total of 256 tokens. We hope our work can inspire future research on integrated multi-modal LLM architectures with native visual generation capabilities. Figure 3: Different paradigms incorporating diffusion and autoregressive models for vision generation. (a) uses patch-level diffusion during every single autoregressive step to tackle continuous outputs [ 27 , 28 , 29 ] ; (b) DART [ 30 ] denoises a full image per every autoregressive step by AR transformer, together inputted with history denoised images; (c) use a single set of continuous outputs to different diffusion steps and require diffusion gradients to train AR models [ 31 , 32 , 33 , 34 ] ; (d) ours uses vanilla AR models, which can train with discrete inputs/outputs by simply cross entropy, and sequentially decode output tokens with our diffusion tokenizer. 2 Related Work 2.1 Diffusion and Autoregressive Models Diffusion models and autoregressive models are currently two main streams of modern generative modeling. Diffusion models [ 20 , 19 , 21 , 35 ] , exemplified by several commercial text-to-image models [ 23 , 22 ] , excel in generating high-quality visual content by iteratively denoising a sample from an initial noise. Though powerful in generating visually pleasing images, the diffusion process typically operates in a dense manner and requires significant sampling steps, which can be computationally expensive. Recent success in language modeling using autoregressive paradigm, especially large language models [ 3 , 1 , 36 , 37 ] , has inspired researchers to explore the potential of this paradigm in visual generation tasks due to its scalability and mature training and inference infrastructures. However, this adaptation raises several challenges, since images are not inherently discrete and linear structures like text. To this end, researchers use vector quantized autoencoders to quantize images into discrete latent codes [ 38 , 7 ] and use raster-scan order to model the image sequence [ 8 , 10 ] . Researchers have also found that image sequence ordering can be defined in various ways [ 14 , 15 , 17 , 16 ] , and the next-token prediction paradigm should be adapted to suit vision modeling accordingly. Though sorts of visual autoregressive models have been proposed,