X2I: Seamless Integration of Multimodal Understanding into Diffusion Transformer via Attention Distillation

Jian Ma 🖂, Qirong Peng, Xu Guo, Chen Chen 🖂, Haonan Lu, Zhenyu Yang · 2025

Read X2I: Seamless Integration of Multimodal Understanding into Diffusion Transformer via Attention Distillation by Jian Ma 🖂, Qirong Peng, Xu Guo, Chen Chen 🖂, Haonan Lu, Zhenyu Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06134.txt.

Introduction

Jian Ma and co-authors give image-making models the power to grasp many kinds of input, not just text. Image models draw well but take orders only in words, while big multimodal models understand pictures, sound, and more. Their 2025 framework, called X2I, moves that understanding into an image model.

A teaching method pulls skill from a teacher model, and a light bridge part links the two sides. Training used only a modest English text set and a short run of computer time. The result handles text in many languages, screen shots, pictures, video, and audio as drawing prompts. Scores stayed within one percent of the teacher on its own tasks. Code and model files were shared in public.

Catalog record

CREATORJian Ma 🖂, Qirong Peng, Xu Guo, Chen Chen 🖂, Haonan Lu, Zhenyu Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06134

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

X2I: Seamless Integration of Multimodal Understanding into Diffusion Transformer via Attention Distillation Jian Ma 🖂, Qirong Peng, Xu Guo, Chen Chen 🖂, Haonan Lu, Zhenyu Yang Abstract Text-to-image (T2I) models are well known for their ability to produce highly realistic images, while multimodal large language models (MLLMs) are renowned for their proficiency in understanding and integrating multiple modalities. However, currently there is no straightforward and efficient framework to transfer the multimodal comprehension abilities of MLLMs to T2I models to enable them to understand multimodal inputs. In this paper, we propose the X2I framework, which endows Diffusion Transformer (DiT) models with the capability to comprehend various modalities, including multilingual text, screenshot documents, images, videos, and audio. X2I is trained on a 100K English corpus in 160 GPU hours. Building on the DiT teacher model, we adopt an innovative distillation method to extract the inference capabilities of the teacher model and design a lightweight AlignNet structure to serve as an intermediate bridge. Compared to the teacher model, X2I shows a decrease in performance degradation of less than 1% while gaining various multimodal understanding abilities, including multilingual to image, image to image, image-text to image, video to image, audio to image, and utilizing creative fusion to enhance imagery. Furthermore, it is applicable for LoRA training in the context of image-text to image generation, filling a void in the industry in this area. We further design a simple LightControl to enhance the fidelity of instructional image editing. Finally, extensive experiments demonstrate the effectiveness, efficiency, multifunctionality, and transferability of our X2I. The open-source code and checkpoints for X2I can be found at the following link: https://github.com/OPPO-Mente-Lab/X2I. 1 Introduction Figure 1 : The primary applications of X2I include multilingual text-to-image, image-to-image, image-text-to-image, video-to-image, audio-to-image, and various multimodal combinations for image generation. The red boxes embedded in the images indicate the input images, text, video, or audio. The red box text below the images represents the abbreviated application names, with the initial letters of different modalities in parentheses. The descriptions below some of the images provide a brief overview of the input audio or video. The recently open-sourced T2I models [ 50 , 61 , 62 , 66 , 54 ] , such as Flux.1 [ 32 ] , have ushered in a new era of AI art due to their ability to generate realistic, photograde images that are both interesting and creative. The framework of T2I models has evolved from early GAN-based models [ 27 , 16 ] , auto-regressive models [ 60 , 85 ] , and UNet-based diffusion models [ 54 ] to DiT models [ 8 , 17 , 71 ] . To enhance both controllability and practicality in visual generation, numerous implicit [ 46 , 41 , 48 ] and explicit [ 47 , 34 , 88 , 84 ] instruction-based editing models leveraging reference images have been developed quickly. Additionally, several unified image instruction editing models [ 22 , 68 , 86 ] have appeared. They all share a critical characteristic: the requirement of collecting comprehensive instruction-based editing datasets and committing to costly training resources. The text encoder in T2I is critical for generating semantically relevant images, as seen from early models such as CLIP [ 57 ] to later models such as T5 [ 58 ] and advanced LLMs [ 72 , 13 ] . The main goal is to improve the semantic understanding of text without considering the impact of other modalities on visual output. In contrast, VLMs, including early approaches like Flamingo [ 2 ] and BLIP2 [ 35 ] , along with newer models like Llama-Vision [ 12 ] , QwenVL [ 3 ] , and InternVL [ 10 ] , have effectively integrated pre-trained visual encoders with LLMs for enhanced text and visual comprehension. Is there a more streamlined way to transfer VLM or MLLM capabilities to DiT to enable diverse modal inputs? The industry is primarily tackling this challenge through two strategies. One method, similar to PEA-Diffusion [ 45 ] , utilizes feature distillation by aligning multilingual LLM encoders to the original dimensions using MLP. However, it requires substantial resources of 1600 GPU hours and large-scale image-text datasets. This distillation relies on the U-Net [ 63 ] block’s outputs and restricts support for multimodal input. Another method is the VLMs-based replacement of T2I [ 43 ] , but it is costly, requiring extensive training or fine-tuning of T2I or MLLMs, necessitating substantial data quality and quantity [ 68 , 22 ] . Alternatively, a solution akin to GlueGen [ 55 ] could be employed to align the features of a single-modal or multimodal encoder with the latent space of existing T2I models. However, aligning solely through the encoder still results in suboptimal overall performance. In this work, we introduce an efficient framework, X2I, along with a simple LightControl module to facilitate the enhancement of image transition from weak to strong fidelity. Utilizing a DiT-based image generation model as a teacher, the student model mirrors the entire architecture of the teacher model but substitutes its text encoder with MLLMs. Both models are capable of processing the same linguistic input. To optimize our training data without gathering images, we opt to distill the teacher model’s inference abilities. In T2I models, this necessitates performing distillation during the reverse denoising phase. Furthermore, we efficiently transfer the visual generation capabilities of the teacher model to the student model by ingeniously designing the AlignNet structure and distilling the attention. As a result, after alignment, the student model acquires the multimodal comprehension abilities of MLLMs during image generation. Moreover, we formulate a template input format intended for the student model to effectively interpret instructions from different modalities. Since MLLMs generally have proficiency in understanding the global semantic features of visual content, we develop LightControl to derive structured data from reference images. This enhances visual fidelity and facilitates precise and controllable image editing. After training the X2I, we conduct various experiments to compare the capabilities of T2I and instructional-based image editing. Additionally, we conduct subjective experiments for image generation with various multimodal mixed inputs, including multilingual, image (I2I), image-text (IT2I), video (V2I), and audio (A2I). The results demonstrate that X2I acquires robust multimodal understanding capabilities with less than 1% performance degradation in the T2I generation. Furthermore, for more personalized tasks like I2I or IT2I, we conduct experiments involving the training of traditional LoRA [ 23 ] to fulfill industry requirements, addressing a training gap in LoRA for these purposes. Moreover, to enhance the adaptability of the framework, only the AlignNet parameters within the model are updated, enabling the aligned student model to accommodate a variety of downstream tasks, such as LoRA, ControlNet [ 70 ] , IP-Adapter [ 84 ] , different fine-tuning models, and compression models. Notably, some objective metrics for tasks involving ControlNet and IP-Adapter show improvements over the teacher model. This improvement is due to X2I’s robust support for image conditional information input, which facilitates further extraction of image features. We also perform experimental analyses on training efficiency, attaining 98.2% of the teacher model’s performance with 48K training samples. In summary, our key contributions are as follows. • We introduce a novel method for transferring the understanding capabilities of MLLMs

Read the complete text