Holistic Tokenizer for Autoregressive Image Generation

Anlin Zheng, Haochen Wang, Yucheng Zhao, Weipeng Deng, Tiancai Wang, Xiangyu Zhang, Xiaojuan Qi · 2025

Read Holistic Tokenizer for Autoregressive Image Generation by Anlin Zheng, Haochen Wang, Yucheng Zhao, Weipeng Deng, Tiancai Wang, Xiangyu Zhang, Xiaojuan Qi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02358.txt.

Introduction

Seven authors, led by Anlin Zheng, present Hita in this 2025 paper, a tokenizer that helps machines build images step by step. Autoregressive image models write visual tokens one after another, like words in a sentence. Standard tokenizers chop an image into local patches, so the model starts with little sense of the whole picture.

Hita fixes that with a holistic-to-local scheme. Learnable holistic tokens come first and carry the global shape of the image. Patch tokens follow, adding local detail. Causal attention keeps each step aware of what came before, matching how the model will generate. A light fusion module then steers the flow of information before the tokens are decoded back into pixels. The design aims at images that hang together as a whole, not just as a grid of parts.

Catalog record

CREATORAnlin Zheng, Haochen Wang, Yucheng Zhao, Weipeng Deng, Tiancai Wang, Xiangyu Zhang, Xiaojuan Qi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-02358

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Holistic Tokenizer for Autoregressive Image Generation Anlin Zheng, Haochen Wang, Yucheng Zhao, Weipeng Deng, Tiancai Wang, Xiangyu Zhang, Xiaojuan Qi Abstract Vanilla autoregressive image generation models generate visual tokens step-by-step, limiting their ability to capture holistic relationships among token sequences. Moreover, because most visual tokenizers map local image patches into latent tokens, global information is limited. To address this, we introduce Hita , a novel image tokenizer for autoregressive (AR) image generation. It introduces a holistic-to-local tokenization scheme with learnable holistic queries and local patch tokens. Hita incorporates two key strategies to better align with the AR generation process: 1) arranging a sequential structure with holistic tokens at the beginning, followed by patch-level tokens, and using causal attention to maintain awareness of previous tokens; and 2) adopting a lightweight fusion module before feeding the de-quantized tokens into the decoder to control information flow and prioritize holistic tokens. Extensive experiments show that Hita accelerates the training speed of AR generators and outperforms those trained with vanilla tokenizers, achieving 2.59 FID and 281.9 IS on the ImageNet benchmark. Detailed analysis of the holistic representation highlights its ability to capture global image properties, such as textures, materials, and shapes. Additionally, Hita also demonstrates effectiveness in zero-shot style transfer and image in-painting. The code is available at https://github.com/CVMI-Lab/Hita . 1 Introduction Within an autoregressive (AR) generation paradigm, the field of large language models (LLMs) [ 51 , 9 , 37 , 35 , 4 , 61 , 52 ] has witnessed significant progress in recent years. In particular, using AR generation along with transformer backbones, GPT-style models have exhibited impressive performance [ 31 , 13 , 1 ] , incredible scalability [ 22 , 16 ] , and versatile flexibility [ 35 , 4 ] across various language tasks. Figure 1 : The concept of holistic tokenizer. A set of learnable queries that capture global properties, such as color, texture, material, etc , from pixels, with semantic-level feature injected, is utilized to reconstruct the image along with image patches. Figure 2 : Holistic tokens capture the global style and content information, as demonstrated by their ability to enable (a) style transfer by replacing the holistic tokens of input images with those of reference images, and (b) zero-shot image inpainting using pre-trained AR generation models . Additionally, the introduction of these new latent tokens accelerates the training speed of the GPT-style AR image generation model by a factor of two (c) . The success of autogressive models in language modeling has driven substantial interest in their application to image generation, giving rise to representative works such as DALL-E [ 38 ] , Parti [ 55 , 56 ] , VAR [ 47 ] , and LlamaGen [ 46 ] . In AR-based image generation, VQVAE [ 12 ] is employed to encode images into discrete tokens. These tokens are then raster-scanned into a 1D sequence to train an AR transformer model like Llama [ 48 ] , which predicts the next token in the sequence. During inference stage, the token sequence generated by the AR model is decoded by the VQVAE’s [ 12 ] decoder to produce the synthesized images. While significant advancements have been made, most research efforts have concentrated on refining and scaling model architectures or optimizing learning objectives for either the VQVAE stage [ 46 , 55 , 56 , 58 ] or the AR transformer stage [ 46 , 38 ] . However, in terms of latent space design, existing approaches predominantly rely on patch-level image representations at either single scales [ 56 , 46 ] or multiple scales [ 47 ] , which limits the ability of latent tokens to capture global and semantic-level information. Although these spatially structured tokens are compatible with diffusion models [ 44 , 10 , 19 , 33 , 41 , 32 ] , which predict all tokens in parallel, they introduce challenges for AR-based models. Autoregressive models typically employ causal attention to predict tokens sequentially [ 6 , 49 , 47 , 46 ] . The absence of holistic information makes it difficult for these models to maintain global coherence, leading to limitations in their capabilities and increasing learning difficulties. As shown in Fig. 2 b (LlamaGen [ 46 ] ) with image inpainting, when half of an image is input into the AR model as a prefix prompt, requiring it to generate the remaining part, the completed images do not align well with the original content. This challenge has also been highlighted in works such as [ 11 , 27 , 53 ] , where bi-directional attention mechanisms are introduced to mitigate the issue, albeit with non-trivial modifications to the AR generation process. To address these issues, we introduce Hita, a novel global-to-local image tokenizer for AR image generation. First, we propose using learnable queries to capture the holistic information from the patch-level image embeddings. Meanwhile, a pre-trained foundation model, such as DINOv2 [ 7 ] , is adopted to inject semantics into the learned queries to enhance their representation. Then, we quantize the learned queries and patch-level embeddings using two separate codebooks, resulting in holistic tokens and patch-level tokens. Next, we fuse the de-quantized holistic and patch-level tokens using a novel token fusion module to reconstruct the image. This design can better prioritizes the holistic token and improving both image reconstruction and generation quality. Once Hita is trained, a vanilla autoregressive (AR) transformer model, e.g. Llama [ 48 ] , can be seamlessly incorporated to generate both holistic and image patch tokens sequentially, which can be decoded into image facilitated by Hita. During the AR generation, the AR generator first generates holistic tokens, which encapsulate global image information and serve as a prefix prompt. These holistic tokens then guide the subsequent generation of local image patch tokens, alleviating difficulties and maintaining global coherence for improved generation quality. Our in-depth ablation experiments confirm that the learned holistic tokens effectively capture global image features, such as shape, color, texture, etc , from pixels (see Fig. 2 a ). This is demonstrated by effortlessly enabling style transfer by combining holistic tokens from reference images with patch-level tokens from the source image. Moreover, without task-specific training, the model can be seamlessly applied to image inpainting by encoding the given incomplete image into latent tokens, which serve as prefix sequence prompts to the AR model, requiring it to generate visual tokens for completing the image (see Fig. 2 b ), demonstrating the capability of holistic tokens in ensuring global consistent synthesis. To further validate its effectiveness, we benchmark Hita for AR image generation using the leading Llama-based [ 48 ] AR transformer model. Extensive experiments show that Hita accelerates the convergence speed during training and significantly improves synthesis quality (see Fig. 2 c ). Specifically, the time required to reach an FID of 4.22 is reduced by a factor of 2.1 . Furthermore, our final AR model with 2B parameters achieves an FID of 2.59 on the ImageNet 256×256 class-conditional image generation benchmark. This outperforms the well-known AR generation model LlamaGen [ 46 ] with 3B parameters. It also surpass the popular diffusion model LDM-4 [ 41 ] by 0.9 FID and 27.1 IS, respectively. In summary,

Read the complete text