UniCompress: Token Compression for Unified Vision–Language Understanding and Generation

Ziyao Wang Chen Chen Jingtao Li Weiming Zhuang Jiabo Huang, Ang Li Lingjuan Lyu · 2026

Read UniCompress: Token Compression for Unified Vision–Language Understanding and Generation by Ziyao Wang Chen Chen Jingtao Li Weiming Zhuang Jiabo Huang, Ang Li Lingjuan Lyu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11320.txt.

Introduction

Ziyao Wang and coauthors compress the visual tokens of unified vision-language models, which process image tokens and text in one autoregressive system. The token volume brings substantial computation and memory overhead, hindering deployment on constrained hardware. Their UniCompress plug-in squeezes and restores tokens under the guidance of learnable global meta tokens, dropping into existing models without full retraining. It cuts image tokens up to fourfold with substantial latency and training-cost gains and only minimal performance loss across both understanding and generation tasks.

Catalog record

CREATORZiyao Wang Chen Chen Jingtao Li Weiming Zhuang Jiabo Huang, Ang Li Lingjuan Lyu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35183
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11320

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UniCompress: Token Compression for Unified Vision–Language Understanding and Generation Ziyao Wang Chen Chen Jingtao Li Weiming Zhuang Jiabo Huang, Ang Li Lingjuan Lyu Abstract Unified models aim to support both understanding and generation by encoding images into discrete tokens and processing them alongside text within a single autoregressive framework. This unified design offers architectural simplicity and cross-modal synergy, which facilitates shared parameterization, consistent training objectives, and seamless transfer between modalities. However, the large number of visual tokens required by such models introduces substantial computation and memory overhead, and this inefficiency directly hinders deployment in resource constrained scenarios such as embodied AI systems. In this work, we propose a unified token compression algorithm UniCompress that significantly reduces visual token count while preserving performance on both image understanding and generation tasks. Our method introduces a plug-in compression and decompression mechanism guided with learnable global meta tokens. The framework is lightweight and modular, enabling efficient integration into existing models without full retraining. Experimental results show that our approach reduces image tokens by up to 4 × 4\times , achieves substantial gains in inference latency and training cost, and incurs only minimal performance degradation, which demonstrates the promise of token-efficient unified modeling for real world multimodal applications. 1 Introduction Recent research on multimodal learning has been moving towards unified models that can understand and generate images within a single autoregressive framework 34 ; 24 ; 42 . A common solution is to encode images into discrete visual tokens produced by a learned tokenizer, then feeds these tokens, together with text tokens, into a language-model backbone. This shared token space enables a broad spectrum of multimodal tasks ( e.g . , image captioning 4 , VQA 39 , image editing 3 ) to be handled by one architecture, simplifying deployment and multi-task training. Figure 1 : We propose UniCompress , a plug-in-and-play token compression algorithm for unified models. The samples are from UniTok 24 . However, a practical limitation of these unified models is token efficiency . Tokenizers from the discrete codebook family ( e.g . , VQ - VAE 27 , the dVAE used in DALL \cdot E 26 , and VQGAN 9 ) often map a 512\times 512 image to 32\times 32=1024 tokens ( i.e . downsampling by a factor of 16 along height and width). Unified models must then use long visual sequences for understanding and adopt equally long sequences for generation, increasing memory footprint, training cost, and inference latency. Sharing a single visual tokenizer across understanding and generation 24 ; 16 reduces engineering complexity but does not reduce the sequence-length. A straightforward solution is to compress the visual tokens. However, experimental results demonstrate that naïve downsampling or uniform token pruning, while effective for image understanding, significantly degrades generation performance by more than 15% . This sets the main challenge since image generation relies on fine-grained, spatially consistent tokens to accurately reconstruct details, hence is sensitive to pruning. Another challenge is ensuring low-cost. While we can train another tokenizer with better token efficiency (e.g. one-D tokenizer) to replace the existing one, but doing so often requires downstream language model training from scratch, which is costly. A more practical approach should be modular, allowing seamless integration with existing tokenizers without the need for full retraining. Thus, the secondary challenge is how to develop a plugin-based compression method compatible with any tokenizer, avoiding expensive LLM retraining. To address the challenges, we introduce UniCompress , a plug-in token compression framework that reduces visual tokens by up to 4\times while maintaining quality for both understanding and generation. Motivated by 15 ; 12 , UniCompress inserts two lightweight modules around an off-the-shelf discrete tokenizer: (i) a compressor that converts a dense H\times W token grid into a short sequence of compressed tokens augmented by a small set of learnable global meta tokens capturing holistic semantics; and (ii) a global-guided decompressor that reconstructs high-fidelity token grids conditioned on those global tokens, restoring long-range structure. We first train the tokenizer together with the compressor and decompressor for image reconstruction task, then freeze the compressed tokenizer and lightly finetune the language model on the compressed tokens for both understanding and generation. The result is a more efficient unifed model with more compact input/output sequence, while retaining the understanding and generation performance through global-guided decompression. 1 1 1 We instantiate a fixed compression ratio for controlled comparisons; the design naturally extends to content-adaptive rates (see § 3 ). Intuitively, the compressed tokens carry salient local evidence, whereas the global meta tokens provide scene-level constraints. During generation, the decompressor uses these global tokens as semantic anchors to autoregressively refine local textures and boundaries, mitigating the detail loss observed with uniform token compression. On standard understanding and generation benchmarks, UniCompress reduces visual tokens by 4\times (for example, 256\rightarrow 64 ) while keeping performance within small margins ( \leq 3-pt drop on understanding; \leq 5-pt FID increase on generation. Relative to uncompressed baselines, UniCompress yields up to 41.8% lower inference latency and 15.4% shorter training time, together with substantial latency savings from shorter sequences. These results indicate a practical path to unified models suitable for resource-constrained platforms. Our key contributions are as follows: • We highlight token efficiency as a bottleneck in unified models and show that naïve token compression disproportionately harms generation . We formalize the objective of a single compact visual token space usable for both understanding and generation. • We propose UniCompress , a plug-in compression framework with global-guided autoregressive decompression. It shortens the visual sequence while preserving generation detail and seamlessly integrates into existing unified models. • We demonstrate strong empirical results across different unified models. Our method achieves up to a 4 \times token reduction while maintaining competitive performance. on both understanding and generation tasks. For both understanding and generation tasks, we cap the performance drop at \leq 5%, and on some benchmarks we fully match pre-compression performance. 2 Related Work Unified Foundation Model. Recent advances in multi-model learning have led to vision foundation models 41 and unified multimodal models 34 ; 24 that support both visual understanding and image generation within a single framework. These models typically encode images into discrete token sequences and process them alongside text using a language model backbone. DreamLLM 8 treats image and text as a joint sequence and leverages autoregressive modeling to seamlessly compose multimodal content. VILA-U 34 moves away from diffusion-based generation, opting instead for a fully token-level decoder that unifies captioning and image synthesis via next-token prediction. To address the limitations of low-capacity tokenizers, UniTok 24 expands the expressiveness of visual tokens using a multi-codebook quantizer, balancing semantic abstraction and reconstruction detail. VARGPT 42 takes a hierarchical approach by predicting both content and resolution scale, enabling controllable and efficient image generation over multiple granularities. Meanwhile, UniFork 21 questions the viability of fully shared backbones and proposes a Y-shaped design that splits deeper layers by task to reduce interference while preserving early fusion. Other works utilize diffusion model as the generation head. For instance, OpenUni 33 and Bagel 7 further push token-level autoregressive generation with diffusion models. To facilitate joint training and architectural simplicity, many of these models adopt a shared tokenizer across modalities, often using ViT-style patch embeddings or VQ-based

Read the complete text