Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLM
Donghwan Chi, Hyomin Kim, Yoonjin Oh, Yongjin Kim, Donghoon Lee, Daejin Jo, Jongmin Kim, Junyeob Baek, Sungjin Ahn, Sungwoong Kim · 2025
Read Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLM by Donghwan Chi, Hyomin Kim, Yoonjin Oh, Yongjin Kim, Donghoon Lee, Daejin Jo, Jongmin Kim, Junyeob Baek, Sungjin Ahn, Sungwoong Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17726.txt.
Introduction
Donghwan Chi and co-authors present a 2025 paper on object-level image tokens for multimodal language models. Such models have to read and generate images, so images need token forms the model handles well. Existing tokenizers capture either global gist or uniform patches. Objects, the level people reason at, fall through the gap. The proposed tokenizer builds on slot attention, which binds features to object slots. A Q-Former style encoder, a diffusion decoder, and residual quantization complete the design.
Tokens now correspond to things in the scene rather than grid squares. AI researchers get finer visual grounding for models that must describe or edit specific objects. Object tokens should help counting, referring, and inpainting tasks where patches blur boundaries. The slot idea imports a classic object-centric learning tool into the language model era. Generation quality at the object level is where current systems most visibly fail.
Catalog record
| CREATOR | Donghwan Chi, Hyomin Kim, Yoonjin Oh, Yongjin Kim, Donghoon Lee, Daejin Jo, Jongmin Kim, Junyeob Baek, Sungjin Ahn, Sungwoong Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17726
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLM Donghwan Chi, Hyomin Kim, Yoonjin Oh, Yongjin Kim, Donghoon Lee, Daejin Jo, Jongmin Kim, Junyeob Baek, Sungjin Ahn, Sungwoong Kim Abstract Recently, multimodal large language models (MLLMs) have emerged as a key approach in achieving artificial general intelligence. In particular, vision-language MLLMs have been developed to generate not only text but also visual outputs from multimodal inputs. This advancement requires efficient image tokens that LLMs can process effectively both in input and output. However, existing image tokenization methods for MLLMs typically capture only global abstract concepts or uniformly segmented image patches, restricting MLLMs’ capability to effectively understand or generate detailed visual content, particularly at the object level. To address this limitation, we propose an object-centric visual tokenizer based on Slot Attention specifically for MLLMs. In particular, based on the Q-Former encoder, diffusion decoder, and residual vector quantization, our proposed discretized slot tokens can encode local visual details while maintaining high-level semantics, and also align with textual data to be integrated seamlessly within a unified next-token prediction framework of LLMs. The resulting Slot-MLLM demonstrates significant performance improvements over baselines with previous visual tokenizers across various vision-language tasks that entail local detailed comprehension and generation. Notably, this work is the first demonstration of the feasibility of object-centric slot attention performed with MLLMs and in-the-wild natural images. 1 Introduction Figure 1: Overview of Slot-MLLM. Slot-MLLM employs Slot Q-Former with Slot Attention to encode images into discrete, object-centric visual tokens. These tokens are treated as a second language within the model, enabling unified autoregressive modeling across multimodal tasks such as image editing (IT2I), visual question answering (I2T), and image generation (T2I). Inspired by the success of LLMs, there has been a rise in ongoing research for multimodal large language models (MLLMs), aiming to leverage strong reasoning and generalization abilities of LLMs due to their extensive background knowledge. For developing such MLLMs, the most efficient way to incorporate non-text modalities into both the input and output of existing LLMs is to construct non-text discrete tokens and perform training, inference, and generation of text as well as non-text tokens within a unified framework identical to that of LLMs. Recently, there have been trials for vision-language MLLMs that utilize discrete image tokens to efficiently process the image modality as a second language within a single, bilingual-like LLM framework [ 14 , 26 , 74 , 52 , 75 ] . For example, in [ 14 ] , an elaborate image tokenizer, called SEED, was proposed for this where causal Q-Former [ 37 ] and diffusion-based decoder [ 58 ] are used to obtain a sequence of image tokens. However, SEED tokens capture only global semantic features of the original image without maintaining the information of local details, and moreover SEED is unable to effectively distribute information across tokens representing different regions within an image. This limitation restricts the corresponding MLLM’s capability in faithfully analyzing or generating local or object-level details of image content. To overcome this limitation, a number of recent works [ 26 , 73 ] have tried to produce a set of local semantic tokens. However, these methods adopt some ad-hoc clustering and merging processes on top of patch-level image features. Indeed, a well-known, more principled approach typically used to distribute object-centric features within an image is Slot Attention [ 44 ] . In this work, we propose to obtain a sequence of object-centric visual tokens for MLLMs through slot attention mechanism that generates image masks in an unsupervised manner via attention competition between queries. In specific, the sequential Q-Former is augmented with slot attention to extract a set of visual tokens from an image where each token represents a distinct region within an image, such as objects, and the modified diffusion decoder reconstructs the original image from the slot visual tokens. Here, slot tokens are discretized by residual vector quantization [ 31 ] for ease of multimodal input and outputs of a unified LLM. In addition, a sequence of slot tokens are trained to be aligned with a sequence of textual inputs using image-text paired data to facilitate the cross-modal understanding and generation in the LLM. The object-centric visual tokens extracted by the proposed Slot Q-Former are then incorporated into both inputs and outputs of the LLM, and the corresponding Slot-MLLM is fine-tuned from the base LLM using a single next-token prediction objective through multimodal pre-training and multimodal instruction tuning. Experimental results on various vision-language multimodal benchmarks show that the proposed Slot-MLLM with Slot Q-Former performs better than the baseline MLLM trained under the same conditions with existing visual tokenizers. This performance improvement is particularly prominent on object-centric image understanding and generation tasks including text-based image editing. Importantly, to our knowledge, this is the first work demonstrating the effectiveness of slot attention for large-scale multimodal training with LLMs using real-world images. To summarize, our main contributions are: (1) an object-centric discretized visual tokenizer by Slot Attention for generative MLLMs, Slot Q-Former; (2) an image-text alignment training of Slot Q-Former to ensure that slot tokens are aligned with texts, enabling efficient MLLM training using the unified next-token prediction; and (3) an empirical validation of Slot-MLLM that outperforms baselines due to the use of Slot Q-Former, which is the first-time realization of Slot Attention performed in large-scale LLMs with diverse natural images. 2 Method The primary goal of this work is to build a generative MLLM that integrates visual and language modalities, ensuring effective object-level control in both image understanding and generation tasks. To achieve this, we propose an object-centric visual tokenizer, named Slot Q-Former, which transforms visual inputs into discrete visual tokens capturing distinct spatial and object-level details of an image. Then, we develop Slot-MLLM, an MLLM that integrates our slot visual tokens and text tokens and performs the multimodal training through a next-token [ 54 , 55 ] prediction for both image understanding and generation tasks. The overall structure of Slot-MLLM with Slot Q-Former is depicted in Figure 1 . 2.1 Object-Centric Slot Q-Former Figure 2: Our tokenizer comprises three main modules: Slot Q-Former, Vector Quantizer, and Visual Decoder. Training occurs in two stages: modules trained in each stage are indicated, and modules exclusive to the second stage—marked in red—include the Vector Quantizer for discrete tokenization. Motivated by the SEED tokenizer [ 14 ] , the proposed visual tokenizer consists of three components: Slot Q-Former, Vector Quantizer, and Visual Decoder, to efficiently transform an input image into a sequence of discrete visual tokens to be used as both inputs and outputs of MLLMs. In particular, we integrate Slot Attention [ 44 ] into the Q-Former architecture [ 37 ] , named Slot Q-Former, to ensure that each slot token encapsulates separated object-level information. Training Slot Q-Former involves two stages. First, Slot Q-Former is trained to extract object-centric information from images. During this stage, the model is trained to reconstruct images using Visual Decoder, simultaneously aligning visual information with text. In the subsequent stage, continuous slot embeddings generated from the first stage is quantized to create discrete tokens suitable for LLM training. Importantly, Slot Q-Former is trained independently from the LLM. This approach enables the encoding of both visual and language data into the same discrete token format,