SNCE: Geometry-Aware Supervision for Scalable Discrete Image Generation

Shufan Li, Jiuxiang Gu, Kangning Liu, Zhe Lin, Aditya Grover, Jason Kuen · 2026

Read SNCE: Geometry-Aware Supervision for Scalable Discrete Image Generation by Shufan Li, Jiuxiang Gu, Kangning Liu, Zhe Lin, Aditya Grover, Jason Kuen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15150.txt.

Introduction

Li and co-authors present SNCE, a training objective for discrete image generators with very large codebooks. Big VQ codebooks improve reconstruction, but models trained on them converge slowly. Instead of a hard one-hot target, SNCE spreads probability over tokens near the true one. Nearness is measured by distance between code embeddings and the image embedding. This teaches the model the geometry of the token space. Across ImageNet generation, text-to-image, and editing tasks, SNCE sped convergence and raised output quality over standard cross-entropy.

Catalog record

CREATORShufan Li, Jiuxiang Gu, Kangning Liu, Zhe Lin, Aditya Grover, Jason Kuen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52197
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15150

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SNCE: Geometry-Aware Supervision for Scalable Discrete Image Generation Shufan Li, Jiuxiang Gu, Kangning Liu, Zhe Lin, Aditya Grover, Jason Kuen Abstract Recent advancements in discrete image generation showed that scaling the VQ codebook size significantly improves reconstruction fidelity. However, training generative models with a large VQ codebook remains challenging, typically requiring larger model size and a longer training schedule. In this work, we propose Stochastic Neighbor Cross Entropy Minimization (SNCE), a novel training objective designed to address the optimization challenges of large-codebook discrete image generators. Instead of supervising the model with a hard one-hot target, SNCE constructs a soft categorical distribution over a set of neighboring tokens. The probability assigned to each token is proportional to the proximity between its code embedding and the ground-truth image embedding, encouraging the model to capture semantically meaningful geometric structure in the quantized embedding space. We conduct extensive experiments across class-conditional ImageNet-256 generation, large-scale text-to-image synthesis, and image editing tasks. Results show that SNCE significantly improves convergence speed and overall generation quality compared to standard cross-entropy objectives. 1 Introduction Modern image generation models achieve high-fidelity synthesis by first encoding raw image pixels into low-dimensional latent embeddings 38 ; 12 ; 62 ; 61 ; 20 . Compared with directly modeling image pixels, training models to generate these latent embeddings has proven to be significantly more effective and scalable. The most widely adopted approach for latent image generation is the latent diffusion model (LDM), which trains a neural network to generate latent image embeddings from i.i.d. Gaussian noise through a continuous diffusion process 42 . Recently, discrete image generation models 15 ; 1 ; 5 have drawn increasing attention due to their compatibility with discrete language-modeling architectures, making them attractive candidates for unified multimodal models 64 ; 23 . In addition, these models demonstrate efficiency advantages because they support key–value (KV) caching during generation 24 ; 31 . Unlike LDMs, which directly learn to generate continuous latent representations, discrete image generation models first discretize continuous image latents into discrete tokens and then train a neural network to generate sequences of such tokens. In general, discrete image generation models can be categorized into two families: autoregressive (AR) models and discrete diffusion models. AR models generate image tokens sequentially in a left-to-right order, whereas discrete diffusion models begin with a sequence consisting entirely of special mask tokens and gradually unmask them to recover clean image tokens. Figure 1 : Limitations of Vanilla Cross Entropy Loss with One-hot Target. Given an input image, vanilla CE loss cannot distinguish between non-closet tokens in the embedding space, even though some of the tokens are close to ground truth in embedding space and can decode to semantically similar images. Addtional details of loss computation in thig figure can be found in appendix. Most discrete image generators, including both AR models and discrete diffusion models, share two common design choices. First, they rely on a discrete image tokenizer that quantizes continuous image latents into discrete tokens. This is typically implemented using a vector quantization (VQ) module. Second, they employ cross-entropy (CE) loss as the training objective to learn a categorical distribution over the vocabulary of possible tokens, although the exact formulation differs slightly between AR models and discrete diffusion models. The quality of the image tokenizer plays a critical role in determining the fidelity of generated images. Several works have shown that larger vocabulary size (codebook size) leads to better reconstruction quality since a larger vocabulary is more expressive and can better capture fine-grained details in the image 48 ; 72 . However, training image generators with a large codebook size can be difficult since it requires larger model size and more data than training image generators with a small codebook. We refer to this challenge as the codebook sparsity problem . As the vocabulary size grows, the frequency of each token during training decreases substantially, resulting in increasingly sparse supervision signals for individual tokens. This can be understood by considering token frequencies in the training data. Consider a dataset of 1 M images where each image is represented by 256 tokens. Assuming uniform token frequencies, each token in the codebook will appear on average 31{,}250 times for an 8{,}192 -sized codebook, but only 1{,}280 times for a 200 K-sized codebook. Hence, the per-token learning signal becomes much sparser as the vocabulary grows, making optimization more difficult. Although this sparsity may appear analogous to language modeling, where the vocabulary size is similarly large, the challenge is fundamentally different because image modeling is inherently a high-entropy problem. For example, when training unified models that generate both images and text 10 , the cross-entropy loss for image tokens ( >7.0 ) is significantly larger than that for language tokens ( <1.0 ). Intuitively, given an English sentence with only the final few words missing, the probability mass typically concentrates on a small set of plausible candidates. In contrast, even when only a small region of an image is missing (e.g., the eye region of a portrait), there exist many plausible pixel configurations that could complete the image. As a result, the prediction distribution in image modeling is inherently more diffuse, making learning with sparse supervision substantially more challenging. A major contributing factor to this issue is that the standard cross-entropy loss uses one-hot probability vectors as training targets, assigning all probability mass to a single ground-truth token while treating all other tokens equally as incorrect. We argue that this formulation is unnatural for tokens produced by a VQ tokenizer. In the VQ process, the quantizer maintains a code embedding for each token in the vocabulary. During tokenization, a continuous image latent embedding is compared with all code embeddings, and the closest one is selected as the ground-truth token according to a similarity metric such as L2 distance or cosine similarity. However, the cross-entropy loss does not distinguish among non-ground-truth tokens: the second-best candidate, the third-best candidate, and a completely unrelated token are all treated identically. This limitation is illustrated in Figure 1 . This issue becomes particularly severe in large-codebook tokenizers, where two highly similar image patches can map to different tokens, making one-hot supervision increasingly brittle. To address the optimization challenges of discrete image generation with large vocabularies, we propose stochastic neighborhood cross-entropy (SNCE) minimization. The key insight of this work is that supervision for discrete image tokens should respect the geometry of the underlying VQ embedding space rather than treating tokens as independent categorical labels. Instead of using one-hot targets corresponding to the nearest codebook entry, SNCE constructs a soft categorical distribution over the vocabulary based on the distances between code embeddings and the encoded image latent. Tokens whose embeddings are closer to the latent representation receive higher probability in the target distribution. This design alleviates the codebook sparsity problem by allowing multiple nearby tokens in the embedding space to receive positive learning signals, rather than supervising the model using only a single closest token. To validate the effectiveness of SNCE, we conduct small-scale experiments on ImageNet-256 and large-scale experiments on text-to-image generation and image editing. Our results show that SNCE significantly improves both convergence speed and final generation fidelity compared with standard CE training. Overall, these findings suggest that incorporating embedding-space

Read the complete text