Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization
Wenhao Zhao [email protected], Qiran Zou [email protected], Rushi Shah [email protected], Yudi Wu [email protected], Zhouhan Lin [email protected], Dianbo Liu [email protected] · 2026
Read Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization by Wenhao Zhao [email protected], Qiran Zou [email protected], Rushi Shah [email protected], Yudi Wu [email protected], Zhouhan Lin [email protected], Dianbo Liu [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17052.txt.
Introduction
Zhao and co-authors investigate collapse problems in vector quantization, a method that turns continuous representations into discrete tokens. Such tokenization underlies many language and diffusion models, yet its failure modes are little studied. Using synthetic and real data, they map two collapse types: tokens collapse and embeddings collapse. They identify triggers such as random initialization and limited encoder capacity. They then propose fixes aimed at each collapse type. The authors call this a first full study of representation collapse in vector quantization.
Catalog record
| CREATOR | Wenhao Zhao [email protected], Qiran Zou [email protected], Rushi Shah [email protected], Yudi Wu [email protected], Zhouhan Lin [email protected], Dianbo Liu [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46949 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17052
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization Wenhao Zhao [email protected], Qiran Zou [email protected], Rushi Shah [email protected], Yudi Wu [email protected], Zhouhan Lin [email protected], Dianbo Liu [email protected] Abstract While discrete tokenizers are suspected to inherently limit sample diversity in token-based generative models, we show this diversity gap is not caused by discretization itself, but rooted in the timing of quantization. In this study, we systematically identify quantization in the initial stage as the primary catalyst for a representational misalignment, where the codebook prematurely shrinks into a narrow latent manifold. This initial shrinkage prevents the codebook from capturing the diverse embedding space of the encoder. Though this may yield deceptively strong reconstructions, it creates a bottleneck that forces the generator to rely on a homogenized set of tokens. Ultimately, the codebook’s failure to anchor to robust representations at the onset of training impairs generative variety and limits sample diversity. To address this, we propose Deferred Quantization, a simple yet effective strategy that introduces a separate continuous learning phase. By allowing the encoder to first establish a well-distributed representation space before introducing discretization, the codebook can effectively anchor to a mature and diverse latent landscape. Across tokenizers and token-based generators, Deferred Quantization consistently reduces shrinkage, improves generative diversity, and preserves reconstruction and compression. We additionally provide a shrinkage diagnostic suite and offer practical guidance for designing diversity-preserving discrete tokenizers. 1 Introduction Figure 1: Token representation shrinkage degrade the diversity of generative model. Vector quantization is a widely used technique to map continuous data into discrete tokens, which assists the transformer-based generative model’s generation. We observe that token representation shrinkage, manifested as narrow distribution in latent space, leads to a shrunk distribution of the generated data. Transformer-based generative models for autoregressive generation have gained significant popularity in recent years in the field of image generation. These models underpin many state-of-the-art systems such as DALL-E ( 3 ) and VAR ( 25 ) , which have found wide-ranging applications in art creation, design automation, and data augmentation. Their practical value lies not only in producing visually compelling images but also in enabling new workflows for creative and industrial domains. Despite their success, transformer-based generative models may exhibit a limitation: the synthetic images they produce tend to cover a narrower distribution than real images. This phenomenon, commonly referred to as mode collapse, results in limited diversity in the generated content. Mode collapse causes the model to ignore valid variations in the data distribution, which limits its generalization, realism, and utility in downstream tasks. In this work, we study on the ability of generative models to produce diverse high-quality outputs from the perspective of tokenization, and study how limitations in diversity arise in token-based generation pipelines. A common implicit assumption in token-based generation is that a tokenizer is good if it yields low reconstruction error. However, reconstruction primarily reflects performance on the high-density regions of the data distribution, while diversity depends critically on broader coverage of the representation space, including low-density regions. This mismatch suggests that a tokenizer can appear strong under reconstruction-centric evaluation, yet still bottleneck downstream diversity. To address mode collapse and improve generative fidelity and diversity, various studies have proposed architectural innovations or alternative training objectives. For example, VQGAN ( 10 ) incorporates vector quantization to learn a discrete codebook, while ImageGPT ( 5 ) treats images as sequences of pixels to better capture complex data distributions. However, these advances do not directly characterize whether the learned discrete tokens sufficiently cover the latent representation space required for diverse generation. In this work, we identify a previously overlooked yet critical aspect of tokenization, which we term token representation shrinkage . It describes a phenomenon in which the learned token embeddings shrink into a small region of the latent space, causing most tokens to cluster around only a few modes, as illustrated in Fig. 1 . When tokens shrink into a limited region of the distribution, the generated outputs are also constrained to a narrow portion of the data space, resulting in reduced diversity and diminished modality coverage. Therefore, token representation shrinkage provides a concrete mechanism by which tokenization can limit downstream diversity even when reconstruction appears satisfactory. Importantly, existing studies ( 32 ; 28 ; 30 ; 22 ) on codebook collapse have mainly focused on the dead-token problem, where some codebook entries are never or rarely used during training, while largely overlooking the issue of insufficient coverage of the embedding space, i.e. , token representation shrinkage. This distinction matters because a codebook may exhibit healthy usage statistics while still occupying only a small region of the latent space, thereby restricting the support available to downstream generative models. To better understand this issue, we analyze its underlying cause and identify a key contributing factor: the commonly used token initialization strategy ( 31 ) in VQ training. In typical practice, token embeddings are initialized from the outputs of an untrained encoder, which already form a compact and clustered distribution in the latent space. This initialization bias restricts the subsequent expansion of the token space during training, making it difficult for the learned tokens to spread out and align with the true data distribution. As a result, it induces token representation shrinkage. We then introduce a simple intervention Deferred Quantization that alleviates token representation shrinkage and enables a controlled study of its impact: we first pretrain the encoder without VQ and then utilize semantic embeddings to initialize the codebook. This decouples representation learning from discretization in the early stage, allowing the encoder to learn meaningful semantic representations before quantization is introduced. As a result, it reduces the resistance faced during VQ optimization and mitigates the shrinkage effect. Using this intervention, we validate the presence and impact of token representation shrinkage through extensive experiments on a variety of synthetic and real-world datasets. Across these settings, we observe that token representation shrinkage is associated with reduced diversity, whereas mitigating shrinkage reliably improves both the diversity and fidelity of generated images. Our main contributions are summarized as follows: • We analyze the detrimental effects of early quantization, demonstrating it causes the codebook to shrink into a narrow, homogenized latent space, which consequently impairs generative sample diversity • We introduce Deferred Quantization, a simple yet effective training intervention that decouples feature learning from discretization. • We validate this Deferred Quantization across synthetic and real-world datasets, showing that mitigating token representation shrinkage consistently restores the codebook’s representational breadth and improves generative variety. 2 Related Works Vector quantization plays an important role in data compression and signal processing under Shannon’s rate–distortion theory ( 12 ; 7 ) . Traditional approaches such as K-means clustering ( 21 ) have been widely used, but they become computationally expensive when applied to high-dimensional data ( 17 ) . To mitigate this challenge, DeepVQ ( 17 ) improved efficiency by mapping data to lower-dimensional latent spaces before quantization. Moreover, ( 26 ) proposed VQ-VAE which integrates VQ with variational autoencoders, using a straight-through estimator ( 2 ) to handle discrete variables. To refine VQ methods for improved performance, variants such as Residual Quantization ( 18 ) , Product Quantization ( 6 ) , and Soft Convex Quantization ( 11 ) further enhanced representation capacity and