Beyond Stationarity: Rethinking Codebook Collapse in Vector Quantization Hao Lu, Onur C. Koyun, Yongxin Guo, Zhengjie Zhu, Abbas Alili, Metin Nafi Gurcan, [4pt] Wake Forest University School of Medicine, Winston-Salem, NC, USA[4pt] {Hao.Lu, Onur.Koyun, Abbas.Alili}@advocatehealth.org{Yongxin.Guo, Zhengjie.Zhu, Metin.Gurcan}@wfusm.edu Abstract Vector Quantization (VQ) underpins many modern generative frameworks such as VQ-VAE, VQ-GAN, and latent diffusion models. Yet, it suffers from the persistent problem of codebook collapse, where a large fraction of code vectors remains unused during training. This work provides a new theoretical explanation by identifying the non-stationary nature of encoder updates as the fundamental cause of this phenomenon. We show that as the encoder drifts, unselected code vectors fail to receive updates and gradually become inactive. To address this, we propose two new methods: Non-Stationary Vector Quantization (NS-VQ), which propagates encoder drift to non-selected codes through a kernel-based rule, and Transformer-based Vector Quantization (TransVQ), which employs a lightweight mapping to adaptively transform the entire codebook while preserving convergence to the k-means solution. Experiments on the CelebA-HQ dataset demonstrate that both methods achieve near-complete codebook utilization and superior reconstruction quality compared to baseline VQ variants, providing a principled and scalable foundation for future VQ-based generative models. The code is available at: https://github.com/CAIR-LAB-WFUSM/NS-VQ-TransVQ.git Keywords: Vector Quantization, Codebook Collapse, Non-stationary Encoder, NS-VQ / TransVQ, Latent Generative Models, Codebook Utilization 1 Introduction (a) A toy example of vanilla VQ in non-stationary process. (b) A toy example of proposed NS-VQ in non-stationary process. (c) A toy example of proposed TransVQ in non-stationary process. Figure 1 : Illustration of codebook adaptation under non-stationary data. (a) In vanilla VQ, codewords fail to track the drifting data distribution, leading to representation collapse. (b) In the proposed NS-VQ, adaptive variance-controlled updates allow the codebook to follow distributional shifts over time, maintaining coverage and stability. (c) In the proposed TransVQ, a few codewords still lag behind the drifting data distribution, but the projector gradients drive all other codewords to move jointly toward the data, improving overall alignment. In the figures, purple dots denote the target distribution Y_{t} (least visible), green dots represent the base data X , blue dots indicate the current batch X_{b} , and red crosses mark the codebook vectors C . Vector Quantization (VQ) has long been a fundamental technique in signal processing, offering a way to represent continuous data by discrete codes 7 . Since its successful integration into Variational Autoencoders (VQ-VAE) by 15 , VQ has become the backbone of many state-of-the-art generative frameworks, including VQ-GAN 16 and Latent Diffusion Models (LDM) 13 . In addition, VQ-based tokenization is widely adopted in visual-language models (VLMs) 17 , where images are discretized into code sequences that can be processed by large language models. Despite these successes, VQ models face a persistent and widely recognized challenge: codebook collapse. As the codebook size increases, a large portion of code vectors remain inactive during training, leading to poor code utilization. This severely limits the effectiveness of VQ in large-scale generative modeling, where scaling the codebook is expected to improve representation power. While numerous practical solutions have been proposed—such as stochastic quantization, codebook reset strategies, distribution regularization, or external feature initialization—most of them are heuristic in nature. They improve utilization rates in practice but lack theoretical justification. Consequently, even when utilization approaches 100%, the final performance across methods varies significantly. This indicates that our current understanding of the codebook collapse problem remains largely empirical and intuition-driven. In this paper, we take a new theoretical perspective by analyzing the non-stationary nature of VQ-VAE. We show that encoder updates inherently make the latent representation a non-stationary process, which explains why codebook entries may fail to receive updates and eventually collapse, (see Fig. 1 ). This connection between non-stationarity and codebook collapse provides, for the first time, a theoretical foundation for understanding the phenomenon. Building on this insight, we propose two new VQ methods designed to address codebook collapse: • Non-Stationary VQ (NS-VQ), illustrated in Fig. 2(a) : introduces a kernel-based approximation that propagates encoder drift to non-selected codes, thereby increasing codebook utilization without breaking theoretical convergence conditions. • Transformer-based VQ (TransVQ), illustrated in Fig. 2(b) : employs a lightweight mapping function P_{\varphi}(\cdot) that adaptively transforms the entire codebook in response to encoder updates. Unlike prior methods such as SimVQ that rely solely on linear mappings and lose convergence guarantees, TransVQ preserves the theoretical conditions for convergence to the k-means solution while ensuring smooth adaptation of all codes. We evaluate these methods on image reconstruction tasks within the VQ-VAE framework. Experimental results show that both NS-VQ and TransVQ significantly improve reconstruction quality, as measured by rFID, LPIPS, and SSIM, while maintaining nearly complete codebook utilization. Furthermore, our analysis of batch-size effects empirically validates the theoretical predictions of our non-stationary framework. Our contributions can be summarized as follows: • We provide a theoretical analysis showing that the non-stationary nature of encoder updates is the root cause of codebook collapse in VQ-VAE. • We propose NS-VQ, a novel variant that propagates encoder drift to non-selected codes via a kernel-based update rule, ensuring improved utilization. • We propose TransVQ, which leverages a learnable codebook mapping function that preserves k-means convergence conditions and effectively adapts the entire codebook. • We conduct extensive experiments on image reconstruction benchmarks, demonstrating that both methods achieve higher utilization and superior reconstruction quality compared to baseline VQ methods. (a) Scheme of the proposed NS-VQ module, which models non-stationary codebook dynamics through variance-controlled updates. (b) Scheme of TransVQ, which introduces transformer-based codebook interaction for non-stationary quantization. Figure 2 : Scheme of the proposed (a) Non-Stationary Vector Quantization (NS-VQ) and (b) the Transformer-based Vector Quantization (TransVQ). 2 Related Work Since its introduction as a stochastic approximation (SA) of k-means 7 and later application to Variational Autoencoders (VQ-VAE) by 15 , Vector Quantization (VQ) has become a fundamental building block in modern visual generative models and image tokenization frameworks. It serves as the core component in models such as VQ-GAN 5 , LDM 13 , and VLM 18 , bridging continuous visual features and discrete token representations 2 . Alongside these applications, a rich body of research has explored improving the optimization and utilization of VQ in neural networks. 2.1 Optimization strategies Several studies have sought to improve the optimization process of VQ models. Huh 8 proposed an alternating optimization scheme that separately updates the encoder and quantization layers to accelerate convergence, albeit with increased computational cost. Baevski et al. (2019) 1 introduced the Gumbel-Softmax trick, providing a differentiable approximation to the discrete argmin operation during codeword selection. More recently, the rotation trick 6 reformulated the mapping from latent features to codebook vectors as a joint rotation-and-scaling transformation, allowing gradient computation through geometric reparameterization. 2.2 Improving codebook utilization Several approaches have been developed to address codebook collapse. Yu 16 proposed maintaining the codebook in a lower-dimensional latent space to increase utilization. SoundStream 19 introduced a dynamic replacement policy that substitutes rarely used codes with randomly sampled vectors from the current batch. SimVQ 21 ; 20 reparametrized the codebook as a linear transformation CW, enabling gradients to flow to non-winner code vectors and achieving high utilization with a simple design. Other works improve utilization through additional constraints, such as orthogonality regularization 14 and cosine-similarity penalties 16 . In contrast, some methods move away from the stochastic k-means formulation entirely—for instance, FSQ 11 , which performs scale-based quantization, and LFQ 17 , which employs directional (sign-based) clustering. 2.3 Alternative quantization approaches Beyond codebook optimization, extensions such as residual VQ 19 aim to reduce quantization error by applying multi-stage residual encoding. Our work differs in focus: rather than introducing heuristic improvements, we theoretically analyze the cause of codebook collapse and propose two methods—NS-VQ and TransVQ—that preserve the convergence property of VQ to the k-means solution. SimVQ inspires TransVQ, yet we extend it with a theoretical explanation of why codebook mapping mitigates collapse and modify it to maintain k-means convergence. Finally, although our experiments employ the standard straight-through estimator (STE), both proposed methods are fully compatible with the Gumbel-Softmax and rotation-trick frameworks. 3 Method 3.1 Preliminaries A Vector-Quantized Variational Autoencoder (VQ-VAE) is a reconstructive encoder–decoder architecture that introduces a vector quantization (VQ) layer to convert continuous latent representations into discrete codes. We assume that data x\in R^{d} is sampled from dataset X=\{x_{1},…,x_{N}\} , Let set the encoder of the VQ-VAE is denoted to E_{\theta}(x)\in\mathbb{R}^{d} , and C=(c_{1},…,c_{K})\in(R^{d})^{K} where K is the number of code vectors. We define the Jacobian matrix (linear approximation of encoder changes with respect to parameters) as J_{\theta}(x)=\frac{\partial E_{\theta}(x)}{\partial\theta} The Voronoi partition associated with codebook C is V_{k}(C)=\{x:||x-c_{k}||\leq||x-c_{j}||,\forall j\} For each sample x_{i} , its assigned code index is denoted by q_{i} , where x_{i}\in V_{q_{i}}(C) . Since the quantization step is non-differentiable, the straight-through estimator (STE) 16 is applied to enable gradient flow. Specifically, the quantized representation is expressed as: Q_{e}(x_{i})=E_{\theta}(x_{i})+sg(c_{q_{i}}-E_{\theta}(x_{i})) where Q_{e} is VQ layer output. This yields (\partial Q_{e}(x_{i}))/\partial E(x_{i})=1. The overall training objective of VQ-VAE combines reconstruction and commitment losses: L=\log p(x\mid Q_{e})+\|\operatorname{sg}(E_{\theta}(x))-c_{q}\|_{2}^{2}+\beta\|E_{\theta}(x)-c_{q}\|_{2}^{2} , where the first term is typically the mean squared error (MSE): \log p(x\mid Q_{e})=\|x-g_{\psi}(Q_{e})\|_{2}^{2} where g_{\psi}(\cdot) is the decoder of VAE. Here, the commitment loss encourages the encoder to stay close to the chosen embedding, while the codebook loss ensures that code vectors move toward the encoder outputs. 3.2 Non-stationarity of VQ-VAE and Its Effect on the Quantization Process We assume k-means initialization, i.e., at iteration t=0 , which means initially C^{(0)} fits c_{k}^{(0)}=\mathbb{E}[x\mid x\in V_{k}(C)] , where \mathcal{E} means expectation or mean, so that C^{(0)} is k-means solution of the latent space and it is also the locally optimal (a fixed point) of VQ algorithm (proof is in Supplementary 8 ) if the encoder is frozen. 3.2.1 Non-stationarity in Training In practice, the encoder parameters \theta are updated during training, making E_{\theta^{(t)}}(x) a time-varying, non-stationary random process. This non-stationarity is a key factor in the emergence of “dead” codebook entries in traditional VQ-VAE training. For simplicity, assume batch size = 1 (the reasoning extends naturally to larger batch sizes). When the first sample x_{1} is processed, the encoder parameters are updated by backpropagation. Under the straight-through estimator (STE), \displaystyle\Delta\theta^{(0)}=\frac{\partial L}{\partial c_{q_{1}}^{(0)}}I\frac{\partial E_{\theta}(x_{1})^{(0)}}{\partial\theta^{(0)}} (1) where L is the reconstruction loss. At the same time, the selected codebook entry is updated by the embedding loss: \displaystyle L_{\text{emb}}^{(t)}=\frac{1}{d}||\operatorname{sg}(E_{\theta}(x_{n})^{(t)})-c_{q_{n}}^{(t)}||^{2} (2) leading to \displaystyle c_{q_{m}}^{(1)}=\begin{cases}\displaystyle c_{q_{m}}^{(0)}+\eta_{0}(x_{1}-c_{q_{m}}^{(0)}),&m=1,\\ c_{q_{m}}^{(0)},&m\neq 1\end{cases} (3) Consider the next sample x_{2} , assumed (without loss of generality) to belong to a different Voronoi cell than x_{1} . After the encoder update, \displaystyle E_{\theta^{(1)}}(x_{2})\approx E_{\theta^{(0)}}(x_{2})+J_{\theta^{(0)}}(x_{2})\Delta\theta^{(0)} (4) Since c_{q_{2}}^{(1)} was not updated in the previous step, c_{q_{2}}^{(1)}=c_{q_{2}}^{(1)} However, the Jacobian term may cause E_{\theta^{(1)}}(x_{2}) to fall outside its original Voronoi region V_{q_{2}}(C) . As a result, c_{q_{2}}^{(1)} may never be selected for updates, eventually becoming a “dead” code. This mechanism explains the early codebook collapse phenomenon observed in 8 [Fig. 9]. We further illustrate this with toy examples (in supplementary 11 and 7 ), showing how encoder shifts (translation, scaling, splitting) can cause persistent dead codes. Intuitively, as the encoder parameters evolve, the mapping from inputs to latent space shifts, causing previously used codebook entries to become obsolete. 3.2.2 Effect of Batch Size From this reasoning, a larger batch size should alleviate codebook collapse: in the first few iterations, more code vectors receive updates, increasing codebook utilization. In the extreme case, if batch size equals dataset size, all code vectors would be updated each iteration, eliminating dead codes entirely. We validated this intuition experimentally by testing VQ-VAE reconstruction performance under different batch sizes (see Fig. 4 ). 3.3 Method 1: Non-Stationary Vector Quantization (NS-VQ) We propose Non-Stationary Vector Quantization (NS-VQ) to address the problem of dead codes caused by encoder non-stationarity. NS-VQ modifies the codebook update rule so that unused codes also receive meaningful updates, thereby mitigating early collapse. 3.3.1 Motivation At t step the encoder is updated with input x_{i} , the representation of other samples like x_{j} becomes \displaystyle E_{\theta^{(t)}}(x_{j})=E_{\theta^{(t)}}(x_{j})+\Delta E_{\theta}(x_{j}) (5) where we use a linear approximation to estimate the additional term \displaystyle\Delta E_{\theta}(x_{j})\approx J_{\theta^{(t)}}(x_{j})\Delta\theta^{(0)} (6) may cause E_{\theta^{(t)}}(x_{j}) to fall outside its original Voronoi region, leaving its codebook entry c_{q_{j}} unused. To counter this, we add a corresponding update to the codebook: \displaystyle\Delta c_{q_{j}}=\begin{cases}\displaystyle\frac{\partial L_{\text{emb}}}{\partial c_{q_{j}}},&q_{j}=q_{i},\\[8.0pt] \Delta E(x_{j}),&q_{j}\neq q_{i}.\end{cases} (7) where L_{emb} is the standard embedding loss. 3.3.2 Estimating \Delta E(x) Since only x_{i} is available in the current batch, we approximate \Delta E(x_{j}) for other codes q_{j} using the gradient update from x_{i} . From one gradient descent step: \displaystyle\Delta\theta=-\eta\frac{\partial L(E_{\theta}(x_{i}))}{\partial\theta}=-\eta J_{\theta}(x_{i})^{T}g_{i} (8) where (g_{i}=\frac{\partial L}{\partial E}/(E_{\theta}(x_{i})) Then, for any x_{j} : \displaystyle\Delta E_{\theta}(x_{j}) \displaystyle\approx J_{\theta}(x_{j})\Delta\theta (9) \displaystyle=-\eta J_{\theta}(x_{j})J_{\theta}(x_{i})^{T}g_{i} (10) \displaystyle=-\eta K_{\theta}(x_{j},x_{i})g_{i} (11) where K_{\theta}(x_{j},x_{i})=J_{\theta}(x_{j})J_{\theta}(x_{i})^{T} being the neural tangent kernel (NTK). Using \Delta\theta\approx J_{\theta}(x_{i})^{-1}\Delta E_{\theta}(x_{i}) to approximate \Delta\theta , we can get \displaystyle\Delta E(x_{j})\approx K_{\theta}(x_{j},x_{i})\,K_{\theta}(x_{i},x_{i})^{-1}\,\Delta E(x_{i}) (12) Since computing the NTK is intractable, we approximate it by a Gaussian RBF kernel: \displaystyle k(x_{j},x_{i})=\exp\!\left(-\frac{\|E(x_{j})-E(x_{i})\|^{2}}{2\sigma^{2}}\right) (13) Thus, \displaystyle\Delta E(x_{j})=\exp\!\left(-\frac{\|E(x_{j})-E(x_{i})\|^{2}}{2\sigma^{2}}\right)\Delta E(x_{i}) (14) In practice, we approximate the distance by replacing E(x_{j}) with its code vector c_{q_{j}} , and Interestingly, we found that simple substitution \Delta E(x_{i}) with (x_{i}-c_{q_{j}}) works better than the full NTK-based estimate. \displaystyle\Delta E(x_{j})\approx\exp\!\left(-\frac{\|E(x_{i})-c_{q_{j}}\|^{2}}{2\sigma^{2}}\right)(E(x_{i})-c_{q_{j}}) (15) This leads to a practical and effective update rule: \displaystyle\Delta c_{q}=\begin{cases}\displaystyle\frac{\partial L_{\text{emb}}}{\partial c_{q_{j}}},&q_{j}=q_{i},\\[8.0pt] \exp\!\left(-\frac{\|E(x_{i})-c_{q_{j}}\|^{2}}{2\sigma^{2}}\right)(E(x_{i})-c_{q_{j}}),&q_{j}\neq q_{i}.\end{cases} (16) This update can be derived from an auxiliary embedding loss: \begin{split}L_{emb}^{\prime}&=\|\operatorname{sg}(E(x_{i}))-c_{q_{i}}\|^{2}\\ &\quad+\sum_{q_{j}\neq q_{i}}\exp\!\left(-\frac{\|E(x_{i})-c_{q_{j}}\|^{2}}{2\sigma^{2}}\right)(E(x_{i})-c_{q_{j}})\end{split} (17) where sg(\cdot) is the stop-gradient operator. 3.3.3 Modified STE It is worth noting that we also revisit the update rule in 8 [Eq. (21)], which proposed a “delay-free” correction term for updating c_{q_{i}}^{(0)} based on E(x_{i})^{(1)}-c_{q_{i}}^{(0)} . That method, however, required tuning an additional hyperparameter v , which limits its practical applicability. In our approach, we re-derive a similar correction without the need for extra hyperparameters. The full derivation and proof are provided in the supplementary 9 Shortly, we updated straight-through estimator (STE) for better update c_{q_{i}} as: \displaystyle Q_{e}=E(x_{i})+sg(c_{q_{i}}-E(x_{i}))+\frac{2}{d}c_{q_{i}}-sg(\frac{2}{d}c_{q_{i}}) (18) where d is the multiplication of batch size and patch numbers at latent feature. In summary, compared with traditional VQ, NS-VQ differs in three aspects: • New embedding loss introduces cross-updates for non-selected codes. • Modified STE improves stability under encoder drift. As shown in Fig. 3 , these changes significantly improve reconstruction quality, with further details provided in the next section. 3.4 Method 2: Transformer-based Vector Quantization (TransVQ) We propose a second approach, called Transformer-based Vector Quantization (TransVQ). The idea is simple: if encoder updates add a drift term to feature representations, why not apply a learnable transformation to the codebook as well? 3.4.1 Motivation For any sample x_{j} , the encoder update at iteration t can be written as \displaystyle E_{\theta^{(t+1)}}(x_{j})=E_{\theta^{(t)}}(x_{j})+J_{\theta^{(t)}}(x_{j})\Delta\theta^{(t)} (19) where \displaystyle\Delta E_{\theta^{(t)}}(x_{j})=J_{\theta^{(t)}}(x_{j})\Delta\theta^{(t)} (20) Instead of leaving the codebook fixed, we introduce a mapping function P_{\phi}(\cdot) that transforms the codebook: \displaystyle C^{\prime}=P_{\phi}(C) (21) During training, only \phi is updated, while the base codebook C remains unchanged. After an update on a batch x_{i} , not only is its corresponding transformed code P_{\phi}(c_{q_{i}}) updated, but all codes are simultaneously updated as \displaystyle P_{\phi^{(t+1)}}(c_{q_{j}})=P_{\phi^{(t)}}(c_{q_{j}})+J_{\phi^{(t)}}(c_{q_{j}})\Delta\phi^{(t)} (22) where, \displaystyle J_{\phi}(t)(c_{q_{j}})=\frac{\partial P_{\phi}^{(t)}(c_{q_{j}})}{\partial\phi^{(t)}} (23) 3.4.2 Key Observation In principle, one could explicitly enforce the alignment between encoder and codebook updates by introducing an additional constraint term \displaystyle L_{align}=||J_{\phi(t)}(c_{q_{m}})\Delta\phi^{(t)}-J_{\phi(t)}(x_{m})\Delta\theta^{(t)}||^{2}, (24) but this would dramatically increase computational complexity and is practically infeasible. Interestingly, our experiments reveal that no such explicit constraint is necessary: the standard embedding loss \displaystyle L_{emb}=||sg(E(x_{i}))-c_{q_{i}}||^{2} (25) alone is sufficient to guide P_{\phi}(\cdot) so that nearly all codebook entries remain active. Across all tested variants of P(\cdot) , this implicit alignment consistently improves both codebook utilization and reconstruction performance. 3.4.3 Design of P_{\phi}(\cdot) The mapping P_{\phi}(\cdot) must be sufficiently generalizable to produce a transformed codebook \displaystyle C^{\prime}=P_{\phi}(C)\in\mathbb{R}^{d} (26) without introducing structural constraints that would break the convergence conditions of VQ to k-means (as proven in Supplementary 8 ). Notably, this is the key different between TransVQ and methods from 21 ; 20 . They force C^{\prime}=WC , which adds another condition, making VQ not converge to the k-means solution anymore. A simple example is that we set the dimension of C as 1, then C^{\prime}=wC . Since C is frozen, C\prime is unlikely to be the k-means solution of E(x) . For this reason, we adopt a lightweight transformer block as P_{\phi}(\cdot) : • Each code vector is treated as a token, • Passed through a single-head linear attention layer, • Followed by a small MLP layer. The output defines the transformed codebook C^{\prime} . 4 Experiments Although our approach can be extended to classification and generative tasks involving VQ, in this work we focus on evaluating NS-VQ and TransVQ within the VQ-VAE framework for image reconstruction. Since our goal is to validate the proposed hypotheses, extensive training under varied hyperparameter settings is required. This makes it impractical to conduct experiments on very large-scale datasets such as ImageNet. Instead, we adopt the CelebA-HQ (256×256) dataset, which contains 30,000 high-quality celebrity face images resampled to 256 pixels. Originally introduced by NVIDIA 9 , this dataset is widely used for generative modeling and contains only images without attributes or labels. Additional results on ImageNet are provided in the Supplementary 12 . The encoder and decoder architectures of VQ-VAE follow the design shown in Tab 1 . We employ MSE loss as the reconstruction objective, and evaluate reconstruction quality using rFID, LPIPS, and SSIM metrics. We set batch size as 16, learning rate as 0.0005, use Adam optimizer. For NS-VQ, the hyperparameter 2\sigma^{2} was initialized to 1 and decayed by a factor of 0.9 after each epoch. For TransVQ, we used a single linear transformer layer with one attention head, a model dimension of 256, and an MLP ratio of 2. All models were trained for 40 epochs on the training set, and their performance was evaluated on the validation set. Table 1 : Encoder and decoder architecture of the proposed model. Encoder Decoder Conv2d (3 \!\rightarrow\! 128, k =4, s =2, p =1), LeakyReLU Conv2d (64 \!\rightarrow\! 256, k =3, s =1, p =1), LeakyReLU Conv2d (128 \!\rightarrow\! 256, k =4, s =2, p =1), LeakyReLU ResidualLayer \!\times\! 6 (256 \!\rightarrow\! 256) Conv2d (256 \!\rightarrow\! 256, k =3, s =1, p =1), LeakyReLU LeakyReLU ResidualLayer \!\times\! 6 (256 \!\rightarrow\! 256) ConvTranspose2d (256 \!\rightarrow\! 128, k =4, s =2, p =1), LeakyReLU LeakyReLU ConvTranspose2d (128 \!\rightarrow\! 3, k =4, s =2, p =1), Tanh Conv2d (256 \!\rightarrow\! 64, k =1, s =1), LeakyReLU – 5 Results From Fig. 3 , we observe that NS-VQ and TransVQ do not suffer from the counterintuitive phenomenon seen in VQGAN-FC 16 , where reducing the codebook dimension improves reconstruction performance. Instead, our methods increase codebook utilization as predicted by theory, leading to improved reconstruction quality. Tabs. 2 and 3 provides a detailed comparison of rFID across different codebook sizes and embedding dimensions, along with the corresponding codebook utilization. Tab. 4 further compares rFID, LPIPS, and SSIM for different VQ methods under their optimal codebook configurations. Finally, Fig. 4 shows the rFID curves of the standard VQ-VAE, with the codebook size and dimension fixed at 64, under varying batch sizes. The consistent decrease in rFID with larger batch sizes empirically supports our theoretical analysis. (a) rFID vs. Codebook Size (lower is better). (b) Codebook Usage vs. Codebook Size (higher is better). Figure 3 : Comparison of the proposed NS-VQVAE and TransVQVAE with VQGAN-FC 16 under varying codebook sizes. (a) rFID comparison showing both NS-VQVAE and TransVQVAE consistently reduce reconstruction error compared with VQGAN-FC 16 . (b) Codebook utilization indicating that both proposed methods maintain nearly full codebook usage, effectively preventing codebook collapse. Table 2 : Comparison of different VQ methods with 3D code dimension under varying codebook sizes. Results are reported in terms of reconstruction FID (rFID, lower is better) and codebook usage (higher is better). Method 64 1024 8912 rFID Usage rFID Usage rFID Usage VQGAN-FC 16 24.10 1.00 18.48 1.00 17.57 0.34 VQVAE2 12 53.25 1.00 31.17 0.40 24.24 0.60 VQGAN-EMA 4 60.86 1.00 17.86 1.00 17.93 1.00 SimVQ 4 23.39 1.00 23.35 1.00 21.69 1.00 NS-VQ 24.55 1.00 17.86 1.00 16.89 1.00 TransVQ 24.42 1.00 17.82 1.00 17.35 1.00 Table 3 : Comparison of different VQ methods with 64D code dimension under varying codebook sizes. Results are reported in terms of reconstruction FID (rFID, lower is better) and codebook usage (higher is better). Method 64 1024 8912 rFID Usage rFID Usage rFID Usage VQGAN-FC 16 24.41 0.66 25.77 0.03 26.00 0.00 VQVAE2 12 85.86 0.82 36.91 0.57 25.87 0.28 VQGAN-EMA 4 159.21 1.00 20.81 1.00 19.39 1.00 SimVQ 21 23.40 1.00 17.77 1.00 14.37 1.00 NS-VQ 23.39 1.00 17.44 1.00 14.01 1.00 TransVQ 23.24 1.00 17.60 1.00 13.70 1.00 Table 4 : Detailed comparison of VQ methods at their best-performing configurations (as determined by rFID in Table 2 and Table 3 ). Results are reported in terms of SSIM (higher is better), LPIPS (lower is better), and rFID (lower is better). Method Codebook Size Code Dim SSIM LPIPS rFID VQGAN-FC 16 8912 3 0.93 0.022 17.57 VQVAE2 12 (3D) 8912 3 0.89 0.042 24.24 VQGAN-EMA 4 1024 3 0.92 0.024 17.86 SimVQ 21 8912 64 0.93 0.017 14.37 NS-VQ 8912 64 0.96 0.015 14.01 TransVQ 8912 64 0.94 0.015 13.70 Figure 4 : rFID curves of standard VQ-VAE with codebook size and code dimension fixed at 64, evaluated under different batch sizes. Larger batch sizes lead to lower rFID values, consistent with our theoretical analysis that larger batches provide more stable codebook updates. 6 Discussion In this work, we restrict our comparison to VQ methods that converge to the k-means solution. Many recent variants have deviated from the stochastic approximation of k-means; for example, FSQ 11 performs direct quantization of latent features rather than clustering, while LFQ 18 can be regarded as sign or directional clustering. These methods are therefore excluded from our evaluation. In Eq. 17, we proposed replacing \Delta E(x_{i}) with (x_{i}-c_{q_{j}}) . In the Supplementary 10 provides a derivation for directly using \Delta E(x_{i}) , which also improves traditional VQ in practice and may be of interest for further exploration. We also note certain limitations. NS-VQ, while theoretically grounded, requires tuning the hyperparameter 2\sigma^{2} , which typically must decay with training iterations—making the optimization process more complex. In contrast, TransVQ is easier to tune but introduces additional computational cost. Interestingly, we observed that replacing the transformer block with a two-layer MLP slightly reduces performance but significantly improves speed, which may be useful in practice. Finally, we would like to point out an open question regarding the common practice of VQ initialization and training. Most existing works—including ours—first initialize the codebook using k-means and then perform online updates through the VQ layer until convergence to the k-means solution in the latent space. However, this raises a natural question: if a global k-means step is already performed, why not simply train a conventional autoencoder, freeze the encoder after training, and then apply k-means on the latent features to obtain the codebook? This approach would even reduce the overall computational cost. One possible explanation is that the quantization layer in VQ-VAE inherently prevents the autoencoder from degenerating into an identity mapping, a problem often encountered in standard AEs. Nevertheless, to the best of our knowledge, there has been no theoretical or empirical study that explicitly analyzes this effect. Understanding why the VQ layer improves representation learning over post-hoc k-means clustering represents an important future direction for VQ-VAE research. 7 Conclusion In this work, we revisited vector quantization from a theoretical perspective and identified the non-stationary nature of VQ-VAE as the fundamental cause of codebook collapse. Building on this insight, we proposed two new approaches, NS-VQ and TransVQ, that propagate encoder drift to non-selected codes or adapt the entire codebook through a learnable mapping. Both methods preserve the convergence conditions of VQ while significantly improving codebook utilization and reconstruction quality. Extensive experiments on CelebA-HQ and additional validations demonstrate that NS-VQ and TransVQ consistently outperform standard VQ variants in terms of rFID, LPIPS, and SSIM, while maintaining near-complete codebook usage. Beyond improving empirical performance, our analysis provides a stronger theoretical foundation for understanding and addressing codebook collapse. Looking forward, the proposed methods can generalize beyond image reconstruction to a wide range of generative and multimodal tasks where VQ serves as the discrete interface, including large-scale visual-language models. Our work thus bridges the gap between theory and practice in vector quantization, offering new directions for building more scalable and reliable VQ-based models. Although our methods enhance stability and utilization, they introduce additional hyperparameters (e.g., 2\sigma^{2} and transformer depth) that require careful tuning. Future work could focus on developing adaptive or self-regularizing mechanisms to automatically control these parameters during training. Moreover, integrating NS-VQ and TransVQ into diffusion, autoregressive, and multimodal architectures may further reveal their potential for large-scale generative modeling. Another promising direction is to explore dynamic or hierarchical codebook expansion, allowing the model to flexibly adjust its representational capacity as training progresses. Lastly, a deeper theoretical study of how quantization interacts with gradient flow and representation learning could provide a unified framework for next-generation VQ-based models. Supplementary Material 8 Proof that VQ Converges to the k -Means Solution Without loss of generality, assume data x\in\mathbb{R}^{d} is sampled i.i.d. from a distribution P . The same argument applies to the finite empirical case, replacing integrals by finite sums. We maintain K centroids (or codes) C=(c_{1},\dots,c_{K})\in(\mathbb{R}^{d})^{K} . Given C , define the Voronoi partition: V_{k}(C)=\{x:\|x-c_{k}\|\leq\|x-c_{j}\|,\;\forall j\}. The expected distortion (quantization error) is \displaystyle J(C) \displaystyle=\mathbb{E}_{x\sim P}\!\left[\min_{k}\|x-c_{k}\|^{2}\right] \displaystyle=\sum_{k=1}^{K}\int_{V_{k}(C)}\|x-c_{k}\|^{2}\,dP(x). (A1) The k -means problem (and equivalently, VQ codebook design) seeks to minimize J(C) . Away from measure-zero tie boundaries, the partition does not change under small centroid moves, so we can differentiate J at regular points: \displaystyle\frac{\partial J}{\partial c_{k}}(C) \displaystyle=\frac{\partial}{\partial c_{k}}\sum_{k=1}^{K}\int_{V_{k}(C)}\|x-c_{k}\|^{2}\,dP(x) \displaystyle=-2\int_{V_{k}(C)}(x-c_{k})\,dP(x). (A2) Setting the gradient to zero gives \displaystyle\int_{V_{k}(C)}x\,dP(x) \displaystyle=c_{k}\int_{V_{k}(C)}dP(x), \displaystyle\Longleftrightarrow\;c_{k}=\mathbb{E}[x\mid x\in V_{k}(C)]. (A3) Thus, the stationary points of k -means are those C^{*} for which every centroid equals the conditional mean of its Voronoi cell—the usual M-step in k -means. Stochastic Online Update. Consider the standard “winner-take-all” online VQ update for a sample x_{t}\sim P : \displaystyle k_{t}(x_{t}) \displaystyle=\arg\min_{j}\|x_{t}-c_{j}^{(t)}\|, (A4) \displaystyle c_{k}^{(t+1)} \displaystyle=c_{k}^{(t)}+\eta_{t}\,\mathbf{1}\{k=k_{t}(x_{t})\}(x_{t}-c_{k}^{(t)}), (A5) where \eta_{t} is the step size. Taking conditional expectation given C^{(t)} : \mathbb{E}[c_{k}^{(t+1)}-c_{k}^{(t)}\mid C^{(t)}]=\eta_{t}\,h_{k}(C^{(t)}), (A6) where \displaystyle h_{k}(C) \displaystyle\triangleq\mathbb{E}\!\left[\mathbf{1}\{x\in V_{k}(C)\}(x-c_{k})\right] \displaystyle=\int_{V_{k}(C)}(x-c_{k})\,dP(x). (A7) Comparing Eq. (A7) with Eq. (A2), we have \frac{\partial J}{\partial c_{k}}(C)=-2h_{k}(C). Hence, the mean update direction equals -\tfrac{1}{2}\nabla J(C) , meaning online VQ is a stochastic approximation (SA) to gradient descent of the k -means objective. We can rewrite the recursion in SA form: c_{k}^{(t+1)}=c_{k}^{(t)}+\eta_{t}\big(h_{k}(C^{(t)})+\xi_{t,k}\big), (A8) with a martingale-difference noise term \displaystyle\xi_{t,k} \displaystyle=\mathbf{1}\{x_{t}\in V_{k}(C^{(t)})\}(x_{t}-c_{k}^{(t)}) \displaystyle\quad-h_{k}(C^{(t)}),\quad\mathbb{E}[\xi_{t,k}\mid C^{(t)}]=0. (A9) Convergence Conditions. The following assumptions hold: • Step sizes (Robbins–Monro): \sum_{t}\eta_{t}=\infty , \sum_{t}\eta_{t}^{2}<\infty . • Boundedness: Data have bounded second moment and centroids are projected to a compact set. • Regularity: h(C) is locally Lipschitz away from tie boundaries, which form a measure-zero set. Under these conditions, by standard SA/ODE theory 10 ; 3 , the discrete iterates track the flow of the ODE \dot{C}(t)=h(C(t))=-\tfrac{1}{2}\nabla J(C(t)). (A10) Using J as a Lyapunov function: \displaystyle\frac{d}{dt}J(C(t)) \displaystyle=\langle\nabla J(C(t)),\dot{C}(t)\rangle \displaystyle=-\tfrac{1}{2}\|\nabla J(C(t))\|^{2}\leq 0. (A11) Therefore, trajectories converge to the stationary set \{\nabla J=0\} . From Eq. (A2), any limit point C^{*} satisfies c_{k}^{*}=\mathbb{E}[x\mid x\in V_{k}(C^{*})],\quad\forall k, which is a k -means fixed point (typically a local minimum). Hence, online VQ with Robbins–Monro step sizes converges almost surely to a k -means solution (local optimum). 9 Modified Straight-Through Estimator (STE) In the main text, we revisited the update rule proposed in 8 [Eq. (21)], which introduced a “delay-free” correction term for updating c_{q_{i}}^{(0)} based on E(x_{i})^{(1)}-c_{q_{i}}^{(0)} . That method, however, required tuning an additional hyperparameter v , which limits its practical applicability. Here, we derive a similar correction from first principles without introducing new hyperparameters. 9.1 Revisiting the Embedding Loss In VQ-VAE, the embedding loss at iteration t is L_{\text{emb}}^{(t)}(x_{n},c_{q_{n}})=\frac{1}{d}\left\|\operatorname{sg}\!\big(E(x_{n})^{(t)}\big)-c_{q_{n}}^{(t)}\right\|^{2}, (A16) which encourages c_{q_{n}}^{(t)} to move toward E(x_{n})^{(t)} . After one iteration, the encoder updates according to E(x_{n})^{(t+1)}=E(x_{n})^{(t)}+\lambda\frac{\partial L}{\partial E(x_{n})^{(t)}}. (A17) The natural question arises: why not directly update c_{q_{n}}^{(t)} so that it also moves toward the newly updated encoder output E(x_{n})^{(t+1)} . 9.2 Delay-Free Correction Term In that case, we define L_{\text{emb}}^{(t)^{\prime}}(x_{n},c_{q_{n}})=\frac{1}{d}\left\|\operatorname{sg}\!\big(E(x_{n})^{(t+1)}\big)-c_{q_{n}}^{(t)}\right\|^{2}. (A18) The gradient with respect to c_{q_{n}}^{(t)} becomes \displaystyle\frac{\partial L_{\text{emb}}^{(t)^{\prime}}}{\partial c_{q_{n}}^{(t)}} \displaystyle=\frac{2}{d}\left(E(x_{n})^{(t+1)}-c_{q_{n}}^{(t)}\right) \displaystyle=\frac{2}{d}\left(E(x_{n})^{(t)}-c_{q_{n}}^{(t)}\right)+\frac{2\lambda}{d}\frac{\partial L}{\partial E(x_{n})^{(t)}}. (A19) The first term can be obtained directly from the standard embedding loss L_{\text{emb}}^{(t)} , while the second term can be propagated to the codebook through the straight-through estimator (STE). 9.3 Modified Quantizer with STE Injection Using the STE formulation, the quantized output is Q_{e}=E(x_{n})^{(t)}+\operatorname{sg}\!\big(c_{q_{n}}^{(t)}-E(x_{n})^{(t)}\big). (A20) To inject the additional gradient flow from E(x_{n})^{(t)} into c_{q_{n}}^{(t)} , we modify the quantizer as \displaystyle Q_{e} \displaystyle=E(x_{n})^{(t)}+\operatorname{sg}\!\big(c_{q_{n}}^{(t)}-E(x_{n})^{(t)}\big) \displaystyle\quad+\tfrac{2}{d}c_{q_{n}}^{(t)}-\operatorname{sg}\!\big(\tfrac{2}{d}c_{q_{n}}^{(t)}\big). (A21) Here, Q_{e} denotes the quantization output of the VQ layer. This modification ensures that the gradient of E(x_{n})^{(t)} automatically propagates into c_{q_{n}}^{(t)} , achieving a delay-free correction without introducing any new hyperparameters. 10 Alternative Update Using \Delta E(x) In the main text, we proposed a practical approximation that replaces \Delta E(x_{i}) with (x_{i}-c_{q_{j}}) . Here, we provide the complementary formulation that directly propagates \Delta E(x) to the codebook updates. 10.1 Recap of the Approximation From the NTK-based derivation, we have \Delta E(x)\approx\exp\!\left(-\frac{\|x-x_{i}\|^{2}}{2\sigma^{2}}\right)\,\Delta E(x_{i}). (A12) Since the future sample x is unavailable at the current iteration, we approximate the distance using code vectors: \|x-x_{i}\|^{2}\approx\|c_{q}-x_{i}\|^{2}. (A13) This approximation allows all codebook entries to be updated in parallel without waiting for additional samples. 10.2 STE-Based Injection Interestingly, we can inject \Delta E(x_{i}) into the straight-through estimator (STE) via the stop-gradient trick. Specifically, the quantized embedding can be expressed as \displaystyle Q_{e} \displaystyle=E(x_{i})+\operatorname{sg}\!\big(c_{q_{i}}-E(x_{i})\big) \displaystyle\quad+\sum_{q_{n}\neq q_{i}}\big(w_{n}c_{q_{n}}-\operatorname{sg}(w_{n}c_{q_{n}})\big), (A14) where the weights are defined as w_{n}=\exp\!\left(-\frac{\|c_{q_{n}}-x_{i}\|^{2}}{2\sigma^{2}}\right). (A15) This formulation distributes a fraction of \Delta E(x_{i}) across all non-selected codes, thereby alleviating the dead-code problem by ensuring that each code receives gradient information proportional to its similarity to the current sample. 11 Toy Demos for Non-Stationary Vector Quantization This appendix provides three toy experiments that illustrate how non-stationarity affects vector quantization and how our proposed NS-VQ stabilizes training under dynamic data distributions. 11.1 Toy Demo 1 — Translation Motion Purpose. Compare two codebook-update rules for vector quantization on a simple non-stationary 2D dataset where a global drift variable {\theta} evolves during training: • EMA (vanilla VQ update method) • NS-VQ (our proposed method) Data and Non-Stationarity. \displaystyle X_{\text{init}}\sim\mathcal{N}(0,I_{2}),\quad N=1500,\quad\text{noise scale}=1.0, \displaystyle{\theta}\in\mathbb{R}^{2},\quad{\theta}_{0}=[0,0]. For each mini-batch with indices idxs : X_{b}=X[\texttt{idxs}]+{\theta},\qquad Y_{t}^{(b)}=X_{\text{init}}[\texttt{idxs}]+(10,10). The drift variable is updated by {\theta}\leftarrow{\theta}+r\cdot\mathrm{mean}(Y_{t}^{(b)}-X_{b}),\quad r=0.1. Codebook and Assignment. C\in\mathbb{R}^{16\times 2},\quad C\sim\mathcal{N}(0,1),\quad w_{i}=\arg\min_{k}\|x-c_{k}\|_{2}^{2}. Update Rules. (1) EMA Update. If cluster k has assigned samples in the current batch: \displaystyle\bar{y}_{k} \displaystyle=\frac{1}{|\{i:w_{i}=k\}|}\sum_{i:w_{i}=k}X_{b}^{(i)}, \displaystyle C_{k} \displaystyle\leftarrow(1-\alpha)C_{k}+\alpha\bar{y}_{k},\quad\alpha=0.3. (2) NS-VQ Online Update. Processed sample-by-sample within each mini-batch using temperature \tau and learning rate lr : \displaystyle d_{k} \displaystyle=\|C_{k}-C_{w}\|^{2},\qquad\omega_{k}=\frac{e^{-d_{k}/\tau}}{\sum_{j}e^{-d_{j}/\tau}}, \displaystyle C_{w} \displaystyle\leftarrow C_{w}+\text{lr}(x-C_{w}), \displaystyle C_{k} \displaystyle\leftarrow C_{k}+\text{lr}\,\omega_{k}(x-C_{k}),\quad k\neq w. After each epoch: \tau\leftarrow\tfrac{1}{2}\tau,\qquad\text{lr}\leftarrow 0.9\,\text{lr}. 11.2 Toy Demo 2 — Expansion / Scaling Purpose. Compare two codebook-update rules on a non-stationary 2D dataset where the data undergo a global linear scaling learned online via a matrix A . Data and Target Mapping (Fixed). X_{\text{init}}\sim\mathcal{N}(0,I_{2}),\quad N=1500. Draw M\in\mathbb{R}^{2\times 2} such that \|M\|_{2}>1 and define the fixed targets: Y_{t}=X_{\text{init}}M^{\top}. Learned State and Batch Variables. Initialize A=I . For a mini-batch with indices idxs : \displaystyle X_{b} \displaystyle=X[\texttt{idxs}],\quad Y_{b}=Y_{t}[\texttt{idxs}], \displaystyle AX_{b} \displaystyle=X_{b}A^{\top},\quad E=Y_{b}-AX_{b}. Update of A (exact code): \displaystyle g_{A} \displaystyle=\frac{2}{B}E^{\top}X_{b},\quad\text{(descent direction of $-\nabla_{A}\|Y_{b}-AX_{b}\|_{F}^{2}$)} \displaystyle A \displaystyle\leftarrow A+rg_{A},\quad r=0.1. All other settings are identical to Appendix 11.1 . We also tested a shrink variant by setting \|M\|_{2}<1 . 11.3 Toy Demo 3 — Split (Quadrant Separation) Purpose. Compare the two update rules on a 2D dataset where samples split by sign into opposite quadrants and drift scales by coordinate sign. Data, Target, and Non-Stationarity. \displaystyle X_{\text{init}} \displaystyle\sim\mathcal{N}({\mu},\sigma^{2}I_{2}),\quad{\mu}=[0.5,\,0.5],\quad\sigma=1.0, \displaystyle Y_{t} \displaystyle=X_{\text{init}}+10\cdot\operatorname{sign}(X_{\text{init}}). Initialize drift {\theta}=[0,0] . For each batch: \displaystyle X_{b} \displaystyle=X[\texttt{idxs}], \displaystyle X_{b} \displaystyle\leftarrow X_{b}+\operatorname{sign}(X_{b})\odot{\theta}, \displaystyle{\theta} \displaystyle\leftarrow{\theta}+r\cdot\mathrm{mean}(Y_{t}^{(b)}-X_{b}),\quad r=0.1. For visualization, the full cloud at each step is X_{b}+\operatorname{sign}(X_{b})\odot{\theta}. 12 Results on ImageNet We employ a Taming-style convolutional encoder–decoder architecture without self-attention, consisting of three resolution stages. The VQ-VAE model is trained on ImageNet-1K at a resolution of 256\times 256 using a batch size of 12 and Adam optimizers with \beta_{1}=0.5 and \beta_{2}=0.9 . The base learning rate is set to 4.5\times 10^{-6} . The reconstruction objective combines mean squared error (MSE) and LPIPS loss. All hyperparameter settings for NS-VQ and TransVQ follow those described in Section 4. Each experiment is trained for four epochs. Table 5 : VQ-VAE architecture (Taming-style, attention-free). The encoder and decoder are symmetric with residual connections. Encoder Decoder Conv2d (3 \rightarrow 128, k=4, s=2, p=1), LeakyReLU Conv2d (3 \rightarrow 512, k=3, s=1, p=1), LeakyReLU ResidualBlock \times 2 (128 \rightarrow 128) ResidualBlock \times 2 (512 \rightarrow 512) Conv2d (128 \rightarrow 256, k=4, s=2, p=1), LeakyReLU ConvTranspose2d (512 \rightarrow 256, k=4, s=2, p=1), LeakyReLU ResidualBlock \times 2 (256 \rightarrow 256) ResidualBlock \times 2 (256 \rightarrow 256) Conv2d (256 \rightarrow 512, k=4, s=2, p=1), LeakyReLU ConvTranspose2d (256 \rightarrow 128, k=4, s=2, p=1), LeakyReLU ResidualBlock \times 2 (512 \rightarrow 512) ResidualBlock \times 2 (128 \rightarrow 128) Conv2d (512 \rightarrow 3, k=3, s=1, p=1) \rightarrow quant_conv (1 \times 1, 3 \rightarrow embed_dim=3) post_quant_conv (1 \times 1, embed_dim=3 \rightarrow 3) — Conv2d (128 \rightarrow 3, k=3, s=1, p=1), Tanh Table 6 : Quantitative comparison on ImageNet-1K at 256\times 256 resolution. Lower rFID and LPIPS indicate better reconstruction quality and perceptual fidelity. Method rFID LPIPS VQGAN 5 1.060 0.113 VQGAN-FC 16 0.600 0.087 VQGAN-EMA 12 1.270 0.124 SimVQ 21 0.551 0.088 NS-VQ (ours) 0.561 0.087 TransVQ (ours) 0.432 0.067 13 Qualitative Reconstructions Figure 5 presents side-by-side reconstructions for six methods: ground-truth (GT), vanilla VQ-VAE, EMA-VQ, NS-VQ (ours), SimVQ, and TransVQ (ours). Consistent with the quantitative results in the main paper, both NS-VQ and TransVQ recover finer facial detail and textures while avoiding over-smoothing. In particular, NS-VQ reduces blocky artifacts by maintaining active code utilization, and TransVQ preserves high-frequency cues (hair strands, eye contours) via its codebook transform. (a) Ground truth (GT). (b) Vanilla VQ-VAE. (c) EMA-VQ. (d) NS-VQ (ours). (e) SimVQ. (f) TransVQ (ours). Figure 5 : Qualitative reconstruction comparison. Each strip shows the same set of identities. NS-VQ and TransVQ maintain sharp edges and facial micro-structure while reducing artifacts observed in vanilla and EMA-VQ. This aligns with the quantitative gains in rFID/LPIPS/SSIM reported in the main paper. Figure 6 : Toy Demo Visualization for Vanilla VQ. Each row corresponds to one toy experiment: translation (1st row), shrink (2nd row), split (3rd row), and expansion (4th row). Each row contains five snapshots showing the convergence process across training steps. Figure 7 : Toy Demo Visualization for NS-VQ. Each row corresponds to one toy experiment: translation (1st row), shrink (2nd row), split (3rd row), and expansion (4th row). Each row contains five snapshots showing how NS-VQ maintains stability and codebook utilization during convergence.