Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generation

Ze Wang, Hao Chen, Benran Hu, Jiang Liu, Ximeng Sun, Jialian Wu, Yusheng Su, Xiaodong Yu, Emad Barsoum, Zicheng Liu AMD GenAI · 2025

Read Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generation by Ze Wang, Hao Chen, Benran Hu, Jiang Liu, Ximeng Sun, Jialian Wu, Yusheng Su, Xiaodong Yu, Emad Barsoum, Zicheng Liu AMD GenAI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21022.txt.

Introduction

Ze Wang and nine co-authors at AMD present Instella-T2I, a text-to-image system built on an unusually compact way of encoding pictures. Instead of the usual grid of codebook tokens, each image becomes a short sequence of binary vectors, just 128 tokens for pictures up to 1024 by 1024 pixels. That is up to thirty-two times fewer tokens than standard methods.

The lean encoding keeps fine detail while making both training and image generation much faster. The team trained their models on a single node of eight AMD accelerators with very large batches, finishing within two hundred GPU days. The resulting models rival current image generators on quality. Notably, they did so without private in-house training data or extra refinement stages. The authors offer the approach as a scalable path for cheaper image generation.

Catalog record

CREATORZe Wang, Hao Chen, Benran Hu, Jiang Liu, Ximeng Sun, Jialian Wu, Yusheng Su, Xiaodong Yu, Emad Barsoum, Zicheng Liu AMD GenAI
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-21022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generation Ze Wang, Hao Chen, Benran Hu, Jiang Liu, Ximeng Sun, Jialian Wu, Yusheng Su, Xiaodong Yu, Emad Barsoum, Zicheng Liu AMD GenAI Abstract Figure 1 : Text-to-image diffusion and auto-regressive generation samples in 1D binary latent space. Details about the prompts, models, and inference configurations are listed in Appendix Section A . Image tokenization plays a critical role in reducing the computational demands of modeling high-resolution images, significantly improving the efficiency of image and multimodal understanding and generation. Recent advances in 1D latent spaces have reduced the number of tokens required by eliminating the need for a 2D grid structure. In this paper, we further advance compact discrete image representation by introducing 1D binary image latents. By representing each image as a sequence of binary vectors—rather than using traditional one-hot codebook tokens, our approach preserves high-resolution details while maintaining the compactness of 1D latents. To the best of our knowledge, our text-to-image models are the first to achieve competitive performance in both diffusion and auto-regressive generation using just 128 discrete tokens for images up to 1024 × 1024 1024\times 1024 , demonstrating up to a 32-fold reduction in token numbers compared to standard VQ-VAEs. The proposed 1D binary latent space, coupled with simple model architectures, achieves marked improvements in speed training and inference speed. Our text-to-image models allows for a global batch size of 4096 on a single GPU node with 8 AMD MI300X GPUs, and the training can be completed within 200 GPU days. Our models achieve competitive performance compared to modern image generation models without any in-house private training data or post-training refinements, offering a scalable and efficient alternative to conventional tokenization methods. 1 Introduction Image tokenizers provide compact representations for images, reducing the costs of operating models directly in the high-dimensional raw pixel space Rombach et al. 2022a ; Esser et al. 2021 . With its efficacy in drastically improving the modeling efficiency, image tokenization plays an increasingly crucial role in modern image/multimodal understanding and generation and has been attracting more research attention recently. Modern image tokenizers can generally be summarized as continuous and discrete tokenizers, with the Kullback-Leibler variational auto-encoder (KL-VAE) Kingma 2013 and the vector quantized variational auto-encoder (VQ-VAE) Van Den Oord et al. 2017 ; Razavi et al. 2019 ; Esser et al. 2021 serving as representatives of each type, respectively. The KL-VAE and its variants have shown significant promise in image generation. These models utilize a continuous latent space that effectively encodes rich representations with as few as four channels per image patch Esser et al. 2021 . Meanwhile, the Gaussian distribution of the latent space, imposed by a KL divergence, facilitates a simpler modeling of the latent representations. KL-VAE dominates image tokenization for modern diffusion-based image generation methods Rombach et al. 2022a ; Rombach et al. 2022c ; Chen et al. 2024c ; Podell et al. 2024 ; Peebles and Xie 2023 ; Ma et al. 2024 ; Li et al. 2024a , where progressive denoising of diffusion models can precisely regress latent values and generate visually appealing images (and videos) under various conditions. However, the continuous latent space challenges the applications of KL-VAE in auto-regressive image generation, where the ‘one-shot’ token prediction may not be able to regress the latent values at a desired precision, leading to poor image generation quality requiring further refinements such as diffusion loss Li et al. 2024a ; Fan et al. 2024 . VQ-VAE and its variants Yu et al. 2021 ; Mentzer et al. 2023 ; Wang et al. 2023b ; Tian et al. 2024 ; Kumar et al. 2024 ; Yu et al. 2024a ; Zhao et al. 2024 provide a discrete latent space with a categorical distribution, which is well-suited for modeling with auto-regressive models Li et al. 2023 ; Chang et al. 2022 ; Sun et al. 2024 ; Tian et al. 2024 . This discrete nature makes VQ-VAE particularly appealing for multi-modal image understanding and generation, as it can be seamlessly integrated into large language models (LLMs) Radford et al. 2019 ; Brown et al. 2020 ; OpenAI 2023 ; Touvron et al. 2023 . However, VQ-VAE training is prone to codebook collapse Ramesh et al. 2021 ; Esser et al. 2021 ; Huh et al. 2023 , which requires additional loss terms and careful hyper-parameter tuning to mitigate Zhu et al. 2024 ; Fifty et al. 2024 . More critically, representing each image patch as a single vector selected from a learned latent codebook limits the representation capacity of the latent space Takida et al. 2022 ; Chen et al. 2024a . This constraint necessitates a large number of latent tokens and an expansive codebook to handle high-resolution images Razavi et al. 2019 . For instance, the VQ-VAE model used in EMU3 Wang et al. 2024 employs 4096 tokens to represent a single 1024 \times 1024 image, resulting in an extremely slow generation speed of \sim 5 minutes per image, preventing the scalability to real-world applications. Traditional latent image representations using 2D latent grids allocate uniform capacity across all image patches Van Den Oord et al. 2017 . Recent research Yu et al. 2024b ; Chen et al. 2024a has shown that this gridded approach can introduce significant redundancy, which, in turn, limits the reconstruction quality of regions with complex features. By learning 1D latent space representations Yu et al. 2024c ; Li et al. 2024b ; Zha et al. 2024 ; Chen et al. 2025 , the spatial correspondence between the latent space and the image patches is removed, enabling more compact latent representations that can achieve comparable or even superior reconstruction quality. However, training 1D tokenizers, particularly with discrete latent spaces, can present practical challenges and often requires complex model architectures and training strategies Kim et al. 2025 , such as two-stage training Yu et al. 2024c , due to its training cost and learning difficulty. Additionally, the use of discrete 1D tokenizers for high-resolution image generation remains significantly underexplored. In this paper, we strive to further push the limit of 1D image tokenizers in discrete latent space for fast and high-fidelity image generation with various conditions. We specifically focus on generating higher-resolution images than in previous studies on 1D discrete image tokenizers while utilizing the same or fewer latent tokens. To increase the representational capacity of each discrete image token, we adopt a binary latent space Wang et al. 2023a approach, representing each token as a binary vector rather than a one-hot selection from the codebook. This method allows each latent token to be expressed as a binary combination of elements of a learned codebook, significantly expanding its representational capacity and paving the way for a more compact latent space Yu et al. 2024a . We train the binary 1D image tokenizer to decode images to multiple resolutions given the same image latent, allowing for flexible image generation and high-reolution images to be generated at the same high speed as the lower-resolution ones. For text-to-image generation, we use a pretrained LLM as the text feature encoder and introduce a streamlined but powerful image generation model that jointly processes text tokens extracted from intermediate layers of the

Read the complete text