MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging

Luyuan Zhang ⋆ \star 1 Siyuan Li ⋆ \star 2,3 Zedong Wang 4 Qingsong Xie † \dagger 5 Cheng Tan 6 Anna Wang 2 Yanhao Zhang 5 Chen Chen 5 Haonan Lu 5 Haoqian Wang † \dagger 1 1 Tsinghua University 2 Westlake University 3 Zhejiang University 4 Hong Kong University of Science and Technology 5 OPPO 6 Shanghai AI Lab ⋆ \star Equal contribution. † \dagger Corresponding author. · 2026

Read MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging by Luyuan Zhang ⋆ \star 1 Siyuan Li ⋆ \star 2,3 Zedong Wang 4 Qingsong Xie † \dagger 5 Cheng Tan 6 Anna Wang 2 Yanhao Zhang 5 Chen Chen 5 Haonan Lu 5 Haoqian Wang † \dagger 1 1 Tsinghua University 2 Westlake University 3 Zhejiang University 4 Hong Kong University of Science and Technology 5 OPPO 6 Shanghai AI Lab ⋆ \star Equal contribution. † \dagger Corresponding author. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30904.txt.

Introduction

Zhang and colleagues unify the two families of visual tokenizers for image generation. Continuous models rebuild images faithfully but produce dense, tangled latents poor for semantic control. Discrete models support autoregressive generation but suffer unstable training and codebook collapse. Their MergeTok jointly optimizes both types in one system, using token merging as a bridge. Clustering similar tokens aligns meaning in the continuous branch and imposes group constraints stabilizing the discrete branch. On ImageNet, MergeTok matches strong rivals with lower error at matched token budgets.

Catalog record

CREATORLuyuan Zhang ⋆ \star 1 Siyuan Li ⋆ \star 2,3 Zedong Wang 4 Qingsong Xie † \dagger 5 Cheng Tan 6 Anna Wang 2 Yanhao Zhang 5 Chen Chen 5 Haonan Lu 5 Haoqian Wang † \dagger 1 1 Tsinghua University 2 Westlake University 3 Zhejiang University 4 Hong Kong University of Science and Technology 5 OPPO 6 Shanghai AI Lab ⋆ \star Equal contribution. † \dagger Corresponding author.
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79085
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30904

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging Luyuan Zhang ⋆ \star 1 Siyuan Li ⋆ \star 2,3 Zedong Wang 4 Qingsong Xie † \dagger 5 Cheng Tan 6 Anna Wang 2 Yanhao Zhang 5 Chen Chen 5 Haonan Lu 5 Haoqian Wang † \dagger 1 1 Tsinghua University 2 Westlake University 3 Zhejiang University 4 Hong Kong University of Science and Technology 5 OPPO 6 Shanghai AI Lab ⋆ \star Equal contribution. † \dagger Corresponding author. Abstract Most visual tokenizers for image generation are bifurcated into two families with complementary limitations: continuous VAEs offer high-fidelity reconstruction but suffer from dense, entangled latents that are poorly suited for semantic control, whereas discrete VQ-based models enable autoregressive generation yet struggle with gradient sparsity, unstable training, and codebook collapse. In this work, we introduce MergeTok, a unified tokenizer that jointly optimizes continuous (VAE) and discrete (VQ) tokenizers within a encoder-decoder architecture, leveraging token merging techniques as a semantic bridge. By clustering similar tokens during encoding, MergeTok establishes a structural prior that provides dual supervision signals: (i) it imposes merged-token semantic alignment in the VAE branch, regularizing its latent space toward disentangled, semantic-aware representations; (ii) it derives group-wise constraints, promoting intra-group diversity and inter-group exclusivity that stabilize VQ training. MergeTok shows competitive reconstruction and generation performance on ImageNet-256, with substantially lower rFID than strong VAE and VQ models under matched token budgets, while producing semantically-organized token representations compatible with both autoregressive and diffusion generators. This shows that a single architecture can endow visual tokenizers with robust semantic organization and generator-friendly discreteness. 1 Introduction Image generation has achieved remarkable success with both auto-regressive [ 9 , 42 ] and diffusion-based paradigms [ 23 ] . These advances rely heavily on the underlying visual tokenizer, which is typically implemented as an auto-encoder that maps an image to a sequence of latent features and imposes a latent-space constraint to encode them either as continuous variables or discrete codes. Existing tokenizers are largely divided by the form of this latent constraint. VQ-based tokenizers such as VQ-VAE [ 30 ] quantize encoder outputs into discrete codebook indices, which naturally support sequence modeling with categorical distributions. VAE-based tokenizers, in contrast, map features to a continuous probabilistic latent space and favor reconstruction fidelity together with stable gradient propagation [ 13 , 36 ] . It is widely believed that advances in the encoder and latent design of tokenizers are central to downstream generation, since these components largely determine token efficiency and semantic controllability [ 12 , 14 ] . Figure 1 : (a) Discrete VQ. Features are quantized by nearest-neighbor codebook lookup, but codebook updates can be sparse. (b) Continuous VAE. Features are mapped to continuous Gaussian latents through reparameterization for stable reconstruction. (c) MergeTok. MergeTok adopts a dual-branch design to jointly optimize VAE and VQ tokenization. The VAE branch introduces online token merging to inject semantic structure into continuous latents, while the resulting token-similarity information is used to guide group-aware VQ quantization, improving discrete token learning. Despite recent progress, both paradigms remain limited by structural weaknesses. (a) Continuous VAE-based tokenizers rely on a latent space that lacks intrinsic organization. Under strong compression, the encoder is driven to preserve fine-grained visual information in dense latent representations, where multiple semantic factors become entangled across latent components. As a result, the learned codes are often weakly disentangled and poorly factorized, which restricts controllable generation and makes it difficult to obtain compact representations that are both semantically structured and reconstruction-friendly. (b) discrete VQ tokenizers face a different bottleneck rooted in optimization. Since quantization is non-differentiable, training depends on approximate gradient propagation, which produces sparse and uneven updates to the codebook. Many codebook entries therefore receive little supervision, increasing the risk of collapse and reducing representation efficiency. These issues create a persistent tension among reconstruction fidelity, semantic structure, efficiency, and training stability. To overcome these limitations, we propose MergeTok, a unified tokenizer that jointly optimizes a VAE branch and a VQ branch, as illustrated in Fig. 1 (c). Our key observation is that token merging provides a natural semantic interface between continuous and discrete tokenization: when content-similar tokens are merged into groups, the resulting source map S can be reused as a structural prior for both branches. We exploit this in two complementary ways. First, we impose merged-token alignment to regularize the VAE latent space toward semantic-aware representations. Second, we use the induced clustering to guide group-aware VQ quantization, promoting within-group code diversity and cross-group exclusivity that stabilize codebook learning. Both branches share the encoder and decoder, are trained jointly under a unified objective, and output 256 tokens at the tokenizer interface. ToMe operates entirely inside the tokenizer’s training loop and is invisible to downstream generators, as detailed in Sec. 3.1 . Experiments on ImageNet-256 demonstrate strong rFID and perceptual quality, consistently outperforming strong VAE-Only and VQ-Only baselines. Our contributions are summarized as follows: • A unified dual-branch tokenizer. We propose MergeTok , a tokenizer that jointly optimizes continuous and discrete latents under a single objective. By using token merging as the semantic connection between the two branches, the stability of VAE training and the structural advantages of VQ can reinforce each other in one encoder-decoder framework. • Merge-aware training constraints. We introduce two simple yet effective objectives derived from the merging process: merged-token alignment , which improves semantic structure in the continuous latent space, and group-aware quantization , which stabilizes VQ training and improves codebook utilization. • Granularity-aware merge ratio sampling. We employ discrete merge-ratio sampling during training so that the model is exposed to multiple token granularities. This enables unified tokenization with improved fidelity and efficiency across reconstruction and generation. 2 Related Work Visual Tokenizers for Image Generation. Modern visual tokenizers convert images into discrete or continuous sequences to enable transformer-based generation. On the discrete side, VQ-based methods address gradient sparsity and codebook collapse through improved differentiability and scaling. IBQ [ 26 ] propagates gradients over full code distributions to maintain high code utilization, while LFQ [ 38 ] replaces vector lookups with binary indices to support very large vocabularies. SoftVQ-VAE [ 3 ] adopts soft categorical posteriors for differentiable training and strong compression. On the continuous side, DC-AE [ 4 ] combines residualized latents with staged training to sustain quality at extreme spatial downsampling, and REPA [ 41 ] aligns latent features to visual encoders to stabilize diffusion optimization and improve generative quality. Collectively, these advances expand vocabulary capacity, restore gradient flow, and impose semantic structure on visual representations. Unified Discrete and Continuous Visual Tokenizers. A growing line of work produces both discrete and continuous representations within a single tokenizer to leverage their complementary strengths. Wave-Particle [ 6 ] builds two parallel branches, a continuous VAE branch for reconstruction and a discrete VQ branch for generation, with partial parameter sharing. VAEVQ [ 35 ] introduces a variational prior into codebook learning so that discrete tokens inherit the regularity of continuous latent structure. TokenBridge [ 32 ] approaches unification from the generation side, encoding images into continuous tokens and subsequently discretizing them for autoregressive synthesis. MergeTok shares this goal but uses the token-merging source map as a structural interface coupling both branches through a shared encoder and decoder, serving as

Read the complete text