Layton: Latent Consistency Tokenizer for 1024-pixel Image Reconstruction and Generation by 256 Tokens
Qingsong Xie, Zhao Zhang, Zhe Huang, Yanhao Zhang, Haonan Lu, Zhenyu Yang · 2025
Read Layton: Latent Consistency Tokenizer for 1024-pixel Image Reconstruction and Generation by 256 Tokens by Qingsong Xie, Zhao Zhang, Zhe Huang, Yanhao Zhang, Haonan Lu, Zhenyu Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08377.txt.
Introduction
Turning a photo into tokens a machine can speak is the gate to image-making models, and the toll is steep: sharp photos want floods of tokens. Qingsong Xie and five co-authors cut that toll in 2025. A photo a thousand pixels square is held in just 256 tokens, a sixteen-fold squeeze over the standard codec.
The bridge is the squeezed space of a trained image model. An encoder writes tokens; a decoder, retooled from that model, reads them back in one or two steps, fixing the colour drift a raw borrow would cause. Pixel-true rebuilds at token thrift is the claim. Cheap tokens mean longer, richer image work for the models downstream.
Catalog record
| CREATOR | Qingsong Xie, Zhao Zhang, Zhe Huang, Yanhao Zhang, Haonan Lu, Zhenyu Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08377
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Layton: Latent Consistency Tokenizer for 1024-pixel Image Reconstruction and Generation by 256 Tokens Qingsong Xie, Zhao Zhang, Zhe Huang, Yanhao Zhang, Haonan Lu, Zhenyu Yang Abstract Image tokenization has significantly advanced visual generation and multimodal modeling, particularly when paired with autoregressive models. However, current methods face challenges in balancing efficiency and fidelity: high-resolution image reconstruction either requires an excessive number of tokens or compromises critical details through token reduction. To resolve this, we propose Latent Consistency Tokenizer (Layton) that bridges discrete visual tokens with the compact latent space of pre-trained Latent Diffusion Models (LDMs), enabling efficient representation of 1024×1024 images using only 256 tokens—a 16 × \times compression over VQGAN. Layton integrates a transformer encoder, a quantized codebook, and a latent consistency decoder. Direct application of LDM as the decoder results in color and brightness discrepancies; thus, we convert it to latent consistency decoder, reducing multi-step sampling to 1-2 steps for direct pixel-level supervision. Experiments demonstrate Layton’s superiority in high-fidelity reconstruction, with 10.8 reconstruction Frechet Inception Distance on MSCOCO-2017 5K benchmark for 1024×1024 image reconstruction. We also extend Layton to a text-to-image generation model, LaytonGen, working in autoregression. It achieves 0.73 score on GenEval benchmark, surpassing current state-of-the-art methods. Project homepage: https://github.com/ OPPO-Mente-Lab/Layton 1 Introduction Image tokenizer [ 38 , 48 ] aims to convert images from their raw pixel-based representations into discrete visual tokens, which can then be used to reconstruct the original image through its corresponding decoder. This approach has garnered significant attention due to its crucial role in image generation, particularly in autoregressive models [ 33 , 37 , 40 ] and masked transformers [ 1 , 2 , 4 , 43 ] . A representative approach, VQGAN [ 46 ] learns a codebook to quantize continuous embeddings of a 256 \times 256 image into 256 discrete tokens using a spatial downsampling ratio of 16 \times —a standard configuration in recent works [ 24 , 27 , 32 , 33 ] . When it comes to high-resolution image reconstruction or generation, e.g. , 1024 \times 1024 pixels, requires predicting 4096 tokens, creating substantial challenges in both computational efficiency and model optimization. This lengthy token sequence complicates downstream applications, such as integrating visual tokens into large multimodal models for interleaved text and image understanding and generation. While recent advancements explore token compression through residual codebooks [ 16 , 37 ] and 1D tokenization [ 13 , 48 ] for 256 \times 256 images, these approaches have not substantially addressed the challenges of higher-resolution image reconstruction and generation. Recent latent diffusion models (LDMs) [ 5 , 15 , 26 ] have demonstrated remarkable success in 1024 \times 1024 image generation by operating in low-dimensional latent spaces. This raises a compelling question: Can discrete visual tokens be aligned with the compact latent space of LDMs to leverage their powerful decoders for high-fidelity reconstruction and generation use? In this paper, we introduce a La tent Consistenc y To ke n izer (Layton), comprising a transformer encoder, a quantized vector codebook, and a latent consistency decoder. Our key insight is to align discrete visual tokens with the compact latent space of pre-trained LDMs, enabling efficient representation of 1024 \times 1024 images with only 256 tokens—a 16 \times reduction compared to VQGAN. Inspired by ControlNet [ 50 ] , we first employ a latent diffusion decoder through copying adaptive blocks in LDM with zero convolution connecting copy and raw LDM. During training, Layton is optimized using diffusion objectives with progressive resolution scaling from 512 \times 512 to 1024 \times 1024. However, relying solely on diffusion loss results in reconstructed images with noticeable discrepancies in color and brightness. To address this, we introduce latent consistency models [ 28 , 44 ] to the decoder, converting the multi-step sampling process into one or two steps. It enables more direct supervision and improving reconstruction fidelity. Moreover, we extend Layton to text-to-image generation model (LaytonGen) by training an autoregressive transformer, which efficiently generates these compact token sequences through text-instructed autoregressive prediction. We conducted comprehensive experiments to evaluate Layton on the ImageNet 50K [ 3 ] , MSCOCO-2017 5K [ 21 ] benchmarks. With 256 tokens to reconstruct 1024 \times 1024 pixel images, Layton achieves 10.80 reconstrution Frechet Inception Distance (rFID) score [ 9 ] on MSCOCO-2017, significantly outperforming VQGAN [ 46 ] , TiTok [ 48 ] , LlmaGen [ 33 ] . On ImageNet benchmark, Layton gets 2.78 rFID, surpassing VQGAN and LlamaGen, comparable to TiTok, but with much better Peak Signal-to-Noise Ratio, Structural Similarity Index Measure, and Learned Perceptual Image Patch Similarity than TiTok. For text-to-image task, LaytonGen obtains 0.73 score on GenEval benchmark [ 8 ] , superior to other diffusion and autoregressive models, e.g., 0.62 by SD3 [ 6 ] , 0.56 by HART [ 34 ] , 0.53 by Show-o [ 43 ] , and 0.32 by LlamaGen. Our contributions are threefold: • We propose Layton, an image tokenizer that bridges discrete visual tokens with the latent space of pre-trained LDMs, demonstrating the ability to reconstruct and generate 1024 \times 1024 images with only 256 tokens. • To alleviate color and brightness discrepancies caused by diffusion objective in reconstruction, we integrate latent consistency model into latent diffusion decoder with direct pixel-level supervision through efficient few-step sampling. • Built on Layton, LaytonGen can generate high-quality 1024 \times 1024 images via text-guided autoregressive token prediction. It achieves leading performances on multiple synthesis benchmarks. 2 Related Work 2.1 Image Tokenization Variational Autoencoders (VAEs) [ 14 ] represent a significant advancement in the field by learning to map inputs to a distribution. Building upon this foundation, VQVAEs [ 38 ] learn discrete representations that form a categorical distribution. This process is further improved in VQGAN [ 7 ] , which enhances the training process through the integration of adversarial training techniques. The transformer architecture within autoencoders is explored in ViT-VQGAN [ 46 ] . RQ-VAE [ 16 ] , introduces residual quantization to the VAE framework, recursively quantizes the feature map in a coarse-to-fine manner, allowing for a precise approximation of the feature map with a fixed codebook size. In a different vein, MAGVIT-v2 [ 47 ] , FSQ [ 23 ] , BSQ-ViT [ 53 ] propose lookup-free quantization, presenting an alternative approach that bypasses traditional lookup mechanisms. TiTok [ 48 ] performs 2D-to-1D distillation, compressing the number of tokens used to represent the same image. VILA-U [ 42 ] , TokenFlow [ 27 ] , and TA-TiTok [ 13 ] introduce text supervision to enhance semantic information for discrete tokens. DiVAE [ 32 ] introduces a diffusion decoder, turning single-step decoding into a multistep probabilistic process at the pixel level. However, they can not reconstruct or generate high-resolution image details by small number of tokens. Motivated by LDMs [ 6 , 15 , 26 ] which are able to generate high-resolution, high-quality images, we explore effective method to leverage the pre-trained LDM as decoder to reconstruct high-resolution images. 2.2 Tokenized Image Generation Image tokenization has become a powerful technique for image generation, allowing images to be represented as discrete tokens that can be manipulated and generated using various modeling approaches. Two prominent methodologies in this domain are the masked-transformer style and the autoregressive style.