Query-Kontext: An Unified Multimodal Model for Image Generation and Editing
Yuxin Song, Wenkai Dong, Shizun Wang, Qi Zhang, Song Xue, Tao Yuan, Hu Yang, Haocheng Feng, Hang Zhou, Xinyan Xiao, Jingdong Wang · 2025
Read Query-Kontext: An Unified Multimodal Model for Image Generation and Editing by Yuxin Song, Wenkai Dong, Shizun Wang, Qi Zhang, Song Xue, Tao Yuan, Hu Yang, Haocheng Feng, Hang Zhou, Xinyan Xiao, Jingdong Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-26641.txt.
Introduction
Yuxin Song and co-authors present a joint model for making and editing images in a 2025 paper. Joint models pair a vision-language brain with an image maker. Skill at reading prompts and pointing to image parts often tangles with skill at fine image craft.
Their model, Query-Kontext, passes a mixed bridge of meaning cues and rough image terms between the two parts. The brain then handles reasoning, while the maker focuses on crisp output. Training runs in three stages, from a light maker head to a large pre-trained one. The split aims to keep identity and layout true to the prompt while lifting image detail.
Catalog record
| CREATOR | Yuxin Song, Wenkai Dong, Shizun Wang, Qi Zhang, Song Xue, Tao Yuan, Hu Yang, Haocheng Feng, Hang Zhou, Xinyan Xiao, Jingdong Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-26641
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Query-Kontext: An Unified Multimodal Model for Image Generation and Editing Yuxin Song, Wenkai Dong, Shizun Wang, Qi Zhang, Song Xue, Tao Yuan, Hu Yang, Haocheng Feng, Hang Zhou, Xinyan Xiao, Jingdong Wang Abstract Unified Multimodal Models (UMMs) have demonstrated remarkable performance in text-to-image generation (T2I) and editing (TI2I), whether instantiated as assembled unified frameworks which couple powerful vision-language model (VLM) with diffusion-based generator, or as naive Unified Multimodal Models with an early fusion of understanding and generation modalities. We contend that in current unified frameworks, the crucial capability of multimodal generative reasoning which encompasses instruction understanding, grounding, and image referring for identity preservation and faithful reconstruction, is intrinsically entangled with high-fidelity synthesis. In this work, we introduce Query-Kontext, a novel approach that bridges the VLM and diffusion model via a multimodal “kontext” composed of semantic cues and coarse-grained image conditions encoded from multimodal inputs. This design delegates the complex ability of multimodal generative reasoning to powerful VLM while reserving diffusion model’s role for high-quality visual synthesis. To achieve this, we propose a three-stage progressive training strategy. First, we connect the VLM to a lightweight diffusion head via multimodal kontext tokens to unleash the VLM’s generative reasoning ability. Second, we scale this head to a large, pre-trained diffusion model to enhance visual detail and realism. Finally, we introduce a low-level image encoder to improve image fidelity and perform instruction tuning on downstream tasks. Furthermore, we build a comprehensive data pipeline integrating real, synthetic, and open-source datasets, covering diverse multimodal reference-to-image scenarios, including image generation, instruction-driven editing, customized generation, and multi-subject composition. Experiments show that our approach matches strong unified baselines and even outperforms task-specific state-of-the-art methods in several cases. Figure 1: Showcase of Query-Kontext model on multimodal reference-to-image tasks. 1 Introduction Unified Multimodal Models (UMMs) have recently achieved notable progress in both image generation (T2I) [ 68 , 27 , 12 , 38 , 5 , 26 , 31 , 55 , 66 , 64 , 90 ] and editing (TI2I) [ 6 , 104 , 102 , 54 , 48 , 25 , 45 , 82 , 87 ] . Two prominent design paradigms have emerged from this work. The first assembled unified framework leverages external diffusion transformers, such as MMDiT [ 27 , 89 ] , which are paired with off-the-shelf vision–language models (VLMs) or large language models (LLMs) to provide semantic conditioning. The second paradigm, naive UMMs, integrate generation and understanding more tightly through mixed-modal early-fusion transformers [ 110 , 26 , 75 , 20 , 85 , 76 , 57 ] , where autoregressive modules with strong reasoning ability are jointly trained with diffusion modules specialized in visual synthesis. While these paradigms expand task coverage and streamline deployment, they also entangle multimodal generative reasoning and high-fidelity rendering. Consequently, the unique strengths of VLMs (semantic understanding, grounding, structured reasoning [ 80 , 2 , 3 , 22 , 23 , 100 , 108 , 99 ] ) and diffusion models (photorealistic synthesis and detail fidelity [ 89 , 11 , 62 , 37 , 14 , 13 , 38 ] ) cannot be fully exploited. We identify two sources of this limitation. First, assembled unified frameworks typically use a frozen VLM or LLM as a static feature extractor, narrowing the conditioning signal to only high-level semantics for the diffusion generator. Second, native UMMs force generative reasoning and visual rendering to be optimized jointly, introducing capacity competition and hindering generalization, particularly when tasks demand both fine-grained edits and strong semantic control. While attempts to mitigate these issues through methods like mixture-of-experts (e.g., LlamaFusion [ 71 ] ) or mixture-of-transformers (e.g., BAGEL [ 26 ] ) have been made, they only partially alleviate the tension. In this work, we propose Query-Kontext , an economic ensemble UMM that leverages the multimodal “kontext” composed of semantic and coarse image conditions to cleanly decouple the generative reasoning of VLM from the high-fidelity rendering of diffusion model. To realize this separation, we develop a three-stage progressive training strategy. Stage 1: Bridge the VLM to a lightweight diffusion head through “kontext” tokens. Using parameter-efficient fine-tuning (LoRA) [ 39 ] , we unleash the potential of VLM and steer it toward multimodal generative reasoning skills such as instruction following, spatial grounding, and identity-preserving image referencing. Stage 2: Scale the lightweight head to a well-trained large diffusion model (roughly 10 \times more parameters). We re-align both the text and “kontext” tokens from the VLM to the scaled diffusion model by using text-to-image generation and image-reconstruction objectives. Stage 3: Introduce a low-level image encoder [ 1 ] that injects fine-grained structural and textural cues into the diffusion model while keeping the VLM frozen. This step strengthens identity preservation [ 101 , 83 , 93 , 72 ] and reconstruction fidelity in [ 54 , 102 , 41 , 98 ] challenging editing scenarios. In summary, our contributions are: • We propose Query-Kontext , an economic ensemble UMM that decouples multimodal generative reasoning in VLMs from the high-fidelity visual rendering performed by diffusion models. • We present a three-stage progressive training strategy that progressively aligns the VLM with increasingly capable diffusion generators while amplifying their respective strengths in generative reasoning and visual synthesis. • We present a deliberate dataset curation scheme to collect real, synthetic, and carefully filtered open-source datasets to cover diverse multimodal reference-to-image scenarios. 2 Query-Kontext In this work, we propose Query-Kontext, a unified multimodal model for image generation and editing that delegates multimodal generative reasoning to the VLM while reserving the diffusion model’s capability for high-quality visual synthesis. In Sec 2.1 , we present the architectural design of the Query-Kontext model (Figure 2 ). In Sec 2.2 , we design a three-stage progressive learning strategy and introduce the details of training recipe (Figure 3 ). In Sec 2.3 , we introduce the implementation details of model hyper-parameters and infrastructures. 2.1 Architecture As shown in Figure 2 , Query-Kontext comprises four main components: a Multimodal Large Language Model (MLLM), a connector module, a Multimodal Diffusion Transformer (MMDiT), and a low-level image encoder (VAE). The MLLM is initialized with the Qwen2.5-VL model [ 3 ] , which encodes and fuses multimodal inputs including the text prompt, input image(s), and a set of learnable query tokens. The output is a fixed-length sequence of kontext tokens Q=\{q_{1},\dots,q_{K}\} which serves as coarse image-level conditioning for the diffusion decoder while providing high-level semantic cues. Intuitively, the kontext tokens Q encode what content should appear in the output image (the semantic information from the text prompt) and how the output should incorporate visual cues from the provided input images, as enforced by the training supervision in Sec. 2.2 . The kontext Q and text tokens T are passed through a lightweight connector module to align them with the diffusion model’s latent space. In practice, we concatenate the connector-generated text embeddings with the kontext embeddings, thereby enriching the semantic context available to the diffusion model. We initialize the diffusion model using our in-house MMDiT model and replace its original text encoder with the MLLM ( training details for this alignment are discussed in Sec. 2.2 ). We concatenate the sequence of text T and kontext tokens Q from the MLLM with: (i) the noisy