D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokens

Panpan Wang, Liqiang Niu, Fandong Meng, Jinan Xu, Yufeng Chen, Jie Zhou · 2025

Read D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokens by Panpan Wang, Liqiang Niu, Fandong Meng, Jinan Xu, Yufeng Chen, Jie Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17155.txt.

Introduction

Panpan Wang and co-authors present D2C, a 2025 method for generating images step by step. Popular generators split into two camps: one uses discrete tokens and scales well but gives weaker images, the other uses continuous values for quality but runs slowly. D2C aims to unlock the strengths of both.

The method brings continuous-style quality to autoregressive generation built on discrete tokens. Image quality in these models depends heavily on the tokenizer that turns pictures into tokens. For image generation research, the hybrid design offers a path to sharper results without giving up the flexibility of token-based models.

Catalog record

CREATORPanpan Wang, Liqiang Niu, Fandong Meng, Jinan Xu, Yufeng Chen, Jie Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17155

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokens Panpan Wang, Liqiang Niu, Fandong Meng, Jinan Xu, Yufeng Chen, Jie Zhou Abstract In the domain of image generation, latent-based generative models occupy a dominant status; however, these models rely heavily on image tokenizer. To meet modeling requirements, autoregressive models possessing the characteristics of scalability and flexibility embrace a discrete-valued tokenizer, but face the challenge of poor image generation quality. In contrast, diffusion models take advantage of the continuous-valued tokenizer to achieve better generation quality but are subject to low efficiency and complexity. The existing hybrid models are mainly to compensate for information loss and simplify the diffusion learning process. The potential of merging discrete-valued and continuous-valued tokens in the field of image generation has not yet been explored. In this paper, we propose D2C , a novel two-stage method to enhance model generation capacity. In the first stage, the discrete-valued tokens representing coarse-grained image features are sampled by employing a small discrete-valued generator. Then in the second stage, the continuous-valued tokens representing fine-grained image features are learned conditioned on the discrete token sequence. In addition, we design two kinds of fusion modules for seamless interaction. On the ImageNet-256 benchmark, extensive experiment results validate that our model achieves superior performance compared with several continuous-valued and discrete-valued generative models on the class-conditional image generation tasks. † † affiliationnotice: * † 1 Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University, Beijing, China 2 Pattern Recognition Center, WeChat AI, Tencent Inc, China {panpwang,jaxu,chenyf}@bjtu.edu.cn {poetniu,fandongmeng,withtomzhou}@tencent.com 1 Introduction During these years, discrete-valued generative models have made remarkable progress in the realm of image generation ( Yu et al. 2022 ; Xie et al. 2024 ; Wu et al. 2024 ; Liu et al. 2024 ; Yu et al. 2024a ) . The discrete image tokenizer that converts encoded features of the image into discrete-valued tokens through vector quantization ( Van Den Oord et al. 2017 ; Esser et al. 2021 ) , is one of the particularly important components of autoregressive and masked image generation models. Afterwards, discrete image tokens are applied to various image understanding and generation tasks with the same paradigm as language tokens. Autoregressive model is one of the most popular solutions because of outstanding achievements in the field of linguistics and their own versatility and scalability. Autoregressive image generation models ( Yu et al. 2021 ; Sun et al. 2024 ; Ma et al. 2024 ; Zhang et al. 2024b ; Yu et al. 2024b ; Luo et al. 2024 ; Yu et al. 2024a ) following the next token prediction paradigm have shown great performance in text-to-image generation and class-to-image generation tasks. In addition to this, non-autoregressive approaches ( Chang et al. 2022 ; Li et al. 2023b ; Chang et al. 2023 ; Yu et al. 2024b ; Chen et al. 2024 ) employ masked image models to predict multiple randomly masked tokens with bidirectional attention, which conforms to the bidirectional correlations essence of visual signals and accelerates the sampling speed in model inference process. Concurrently, another predominant methods are continuous-valued generative models, which employ a continuous tokenizer to encode and decode the image without vector quantization ( Kingma 2013 ) . Diffusion models ( Song et al. 2020 ; Ho et al. 2020 ) are the most common methods in continuous-valued generative models, which recover latent space of image from random Gaussian noise through progressively denoising, such as LDM with the U-Net block ( Rombach et al. 2022 ) , DiT with the transformer block ( Peebles & Xie 2023 ) . In addition, methods about merging autoregressive model and diffusion model ( Tschannen et al. 2023 ; Fan et al. 2024 ; Zhou et al. 2024 ; Zhao et al. 2024 ) are beginning to emerge. Although diffusion models achieve slightly better generation performance than discrete-valued generative models, the challenges of low computational efficiency and complicated structure are still unavoidable. Subsequently, MAR model ( Li et al. 2024a ) uses a small MLP network to iteratively denoise with autoregressive prior that achieves the trade-off between accuracy and speed in image generation. However, it does not matter whether discrete-valued generative models or continuous-valued generative models, each still faces some challenges. Firstly, the discrete image tokenizer in discrete-valued generative models reveals poor reconstruction quality because quantizing continuous encoded image features to discrete-valued tokens results in information loss. Secondly, high computational complexity of continuous-valued generative models constrains training and inference speed while the complete model is applied to iteratively denoise on the entire image. Third, research into merging discrete-valued tokens with continuous-valued tokens has not yet been well developed. To address these challenges, we introduce D2C , a simple and effective hybrid model that improves image generation quality through synergizing discrete-valued and continuous-valued tokens. Specifically, it is a two-stage autoregressive framework. In the first phase, we train a small discrete-valued autoregressive model to sample the discrete token sequence as the coarse-grained image features that can be decoded into a low-quality image. In the next phase, we optimize a hybrid autoregressive model to generate continuous-valued tokens as the fine-grained image features conditioned on class token and generated discrete-valued tokens. Then, the decoder of VAE model is employed to convert the generated continuous-valued tokens into a high-quality image as our model final output. For realizing seamless interaction between discrete-valued and continuous-valued tokens, we elaborately design two kinds of fusion modules, cross-attention module and q-former module. Currently, there are a couple of work, such as DisCo-Diff ( Xu et al. 2024 ) , HART ( Tang et al. 2024 ) that are a little similar to our idea. But being different from them, discrete-valued and continuous-valued tokens are generated by different pretrained image tokenizers in our method and the two can be decoded separately, each yielding an image. Besides, our model is a two-stage framework to achieve coarse-grained to fine-grained image generation. To summarize, our contributions are: • We propose a novel two-stage hybrid autoregressive framework, D2C, which enhances image generation quality with fusing the discrete-valued and continuous-valued tokens from the independently trained image tokenizers. • We design two kinds of fusion modules including cross-attention module and q-former module, to achieve seamless interaction between discrete-valued and continuous-valued tokens. • To verify our model effectiveness and versatility, we conduct experiments on the ImageNet-256 benchmark. Experimental results indicate that our model outperforms both previous continuous-valued and discrete-valued generative models on the class-to-image task. Our model is more faster than MAR on this task in inference. 2 Related Work Continuous-valued image generation models mainly contain diffusion models and autoregressive models. Diffusion models ( Ho et al. 2020 ; Rombach et al. 2022 ; Ho et al. 2022 ; Betker et al. 2023 ) are known as state-of-the-art methods due to high-quality visual generation, which generate the image through an iteratively denoising process on the entire image. At first, convolutional U-Net is the backbone architecture in latent diffusion models ( Rombach et al. 2022 ; Podell et al. 2023 ) . To improve scalability and efficiency, the architecture of transformer from Peebles & Xie 2023 ; Bao et al. 2023 ; Chen et al. 2023 is proposed to replace the

Read the complete text