Unified Autoregressive Visual Generation and Understanding with Continuous Tokens

Lijie Fan, Luming Tang, Siyang Qin, Tianhong Li, Xuan Yang, Siyuan Qiao, Andreas Steiner Chen Sun Yuanzhen Li Tao Zhu Michael Rubinstein Michalis Raptis, Deqing Sun Radu Soricut · 2025

Read Unified Autoregressive Visual Generation and Understanding with Continuous Tokens by Lijie Fan, Luming Tang, Siyang Qin, Tianhong Li, Xuan Yang, Siyuan Qiao, Andreas Steiner Chen Sun Yuanzhen Li Tao Zhu Michael Rubinstein Michalis Raptis, Deqing Sun Radu Soricut (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13436.txt.

Introduction

Most image tools split their jobs: some read pictures, others make them, and the two camps rarely share a brain. Lijie Fan and co-authors merge the jobs in 2025 in one model, UniFluid, which writes text as discrete tokens and paints images as smooth, continuous ones. Trained the right way, each skill fed the other instead of competing.

With a careful balance of the two training goals, the single model matched or beat specialists at both tasks. Stronger base language models and shuffled generation order lifted image quality further. The system also handled editing, captioning, and visual questions. The authors present it as proof that one autoregressive model can both see and draw well.

Catalog record

CREATORLijie Fan, Luming Tang, Siyang Qin, Tianhong Li, Xuan Yang, Siyuan Qiao, Andreas Steiner Chen Sun Yuanzhen Li Tao Zhu Michael Rubinstein Michalis Raptis, Deqing Sun Radu Soricut
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13436

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unified Autoregressive Visual Generation and Understanding with Continuous Tokens Lijie Fan, Luming Tang, Siyang Qin, Tianhong Li, Xuan Yang, Siyuan Qiao, Andreas Steiner Chen Sun Yuanzhen Li Tao Zhu Michael Rubinstein Michalis Raptis, Deqing Sun Radu Soricut Abstract We present UniFluid, a unified autoregressive framework for joint visual generation and understanding leveraging continuous visual tokens. Our unified autoregressive architecture processes multimodal image and text inputs, generating discrete tokens for text and continuous tokens for image. We find though there is an inherent trade-off between the image generation and understanding task, a carefully tuned training recipe enables them to improve each other. By selecting an appropriate loss balance weight, the unified model achieves results comparable to or exceeding those of single-task baselines on both tasks. Furthermore, we demonstrate that employing stronger pre-trained LLMs and random-order generation during training is important to achieve high-fidelity image generation within this unified framework. Built upon the Gemma model series, UniFluid exhibits competitive performance across both image generation and understanding, demonstrating strong transferability to various downstream tasks, including image editing for generation, as well as visual captioning and question answering for understanding. Figure 1: Generated images from our UniFluid autoregressive model after aesthetic finetuning. 1 Introduction Large Language Models (LLMs) have recently advanced from text-centric architectures, such as BERT [ 11 ] and GPT [ 35 ] , toward multimodal systems capable of understanding and generating content across different modalities. GPT-3 [ 5 ] and PaLM [ 9 ] show that scaling language models leads to emergent capabilities, while Flamingo [ 2 ] further demonstrates that incorporating visual inputs facilitates unified multimodal reasoning. This trend toward unified vision-language model—using a single model for diverse tasks of visual understanding and generation—has promising potential for leveraging knowledge and reasoning abilities that transfer across different vision and language tasks, ultimately facilitating more robust and generalizable multimodal representation and modeling capabilities. Motivated by the advantages and strong scaling properties of autoregressive models, coupled with their simplicity, we investigate a pure autoregressive framework for unified visual generation and understanding, without the limitations introduced by vector quantization (VQ). In this paper, we introduce UniFluid, a unified framework that leverages continuous visual tokens within an autoregressive architecture to jointly handle vision-language generation and understanding tasks. Building upon pre-trained Gemma [ 48 ] on large-scale text corpus, UniFluid unlocks powerful visual generation and understanding capabilities through training with paired image-text data, and further allows these two tasks to mutually benefit each other within a single architecture. Specifically, UniFluid adopts a unified autoregressive framework where both text and continuous visual inputs are embedded as tokens in the same space, enabling seamless joint training of image generation and understanding tasks. UniFluid integrates a continuous tokenizer [ 12 , 21 ] for image generation and a pre-trained SigLIP [ 59 ] image encoder for understanding tasks, while textual inputs are processed using a standard SentencePiece tokenizer [ 20 ] . The resulting multimodal sequences are modeled autoregressively using Gemma [ 48 ] as the underlying transformer backbone. Task-specific prediction heads—a diffusion-based head for image generation and a cross-entropy head for text generation—ensure effective modality-specific training and inference, enabling UniFluid to efficiently learn shared representations that mutually enhance its generation and understanding capabilities. Our experiments demonstrate several key advantages of the proposed unified training strategy. We find though there is a trade-off between the two tasks, a carefully tuned training recipe can allow the tasks to support each other and outperform the single-task baselines. Effectively balancing the loss between the tasks allows a single model that performs both with results superior to or on par with single-task models. Moreover, the choice of pre-trained LLM backbone significantly impacts visual generation performance. We also find that while employing random generation order is essential for high-quality image synthesis, it is less critical for understanding tasks. Finally, our unified pre-trained models show strong generalization and transferability, achieving compelling results in downstream applications, including image editing and various vision-language understanding benchmarks. 2 Related Works Multimodal Large Language Models . Multimodal Large Language Models [ 2 , 10 , 23 , 26 , 25 , 3 , 42 ] have shown significant performance in visual understanding tasks. Flamingo [ 2 ] adopted a frozen LLM and vision encoder, utilizing perceiver with cross-attention to bridge the modalities. LLaVA [ 26 , 25 ] proposed instruction tuning over pre-trained LLMs with multimodal inputs to align a pre-trained image encoder into the LLM’s embedding space, thereby enabling it with visual understanding and instruction following capabilities. MiniGPT-4 [ 62 ] and mPLUG-Owl [ 56 ] have shown vision encoders can be connected to LLMs through projection layers, demonstrating sophisticated visual reasoning capabilities. The PaliGemma [ 3 , 42 ] series built upon the Gemma [ 47 , 48 ] model family to develop versatile vision-language models capable of strong transfer to diverse downstream visual understanding tasks. Autoregressive Image Generation . While diffusion models [ 41 , 36 ] have achieved impressive success in image generation, autoregressive image generation methods have also shown significant development, driven by their simplicity and closeness to LLM training paradigms. A large body of research centers on tokenizing images into discrete tokens and applying autoregressive objectives to these discrete representations. Notable examples include Parti [ 57 ] and Muse [ 6 ] . [ 49 ] proposes an approach that operates on image scales, progressively refining resolutions from coarse to fine through next-scale prediction. Alternatively, works such as MAR [ 21 ] , Fluid [ 12 ] , and techniques employing per-token diffusion heads on top of LLM-predicted embeddings have explored autoregressive image generation with continuous visual tokens. Unified Multimodal Models . There is growing research interests in unifying visual generation and understanding within a single model. VQ-based models, such as Chameleon [ 46 ] , Emu [ 44 ] , and Janus [ 54 , 8 ] , propose encoding visual inputs into discrete tokens and unifying tasks into next-token prediction within this discrete token space. Models with hybrid training targets, such as Transfusion [ 61 , 38 ] and Show-O [ 55 ] , aim to unify next-token prediction objectives with diffusion objectives within a single framework. MetaMorph [ 50 ] maintains the autoregressive objective by regressing visual SigLIP [ 59 ] features using an LLM, but necessitates a separate diffusion model to decode the predicted latent features into images. Our approach distinguishes itself by performing per-token autoregressive generation using continuous visual tokens. This maintains the next-token prediction objective, while not being limited by the vector quantized tokens. [ 45 ] also explores using continuous tokens to generate multimodal outputs. 3 Method In this section, we illustrate the architecture of our UniFluid model. The model expects both image and text sequences as input and achieves joint training on both generation and understanding tasks, using next-token prediction as its training objective. Figure 2: High-level illustration of UniFluid. UniFluid performs joint training of image generation and understanding tasks through next token prediction. For image embeddings, we use a VAE image tokenizer for generation, and a SigLIP image encoder for understanding. BOI/BOS stands for beginning of Image/Sentence. 3.1 Unified Autoregressive Modeling with Continuous Visual Tokens Our approach leverages the autoregressive paradigm to unify visual understanding

Read the complete text