LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning

Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, Chongxuan Li, Large Models, Intelligent Governance Engineering Research Center of Next-Generation, Intelligent Search, Recommendation, MOE Ant Group · 2025

Read LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning by Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, Chongxuan Li, Large Models, Intelligent Governance Engineering Research Center of Next-Generation, Intelligent Search, Recommendation, MOE Ant Group (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16933.txt.

Introduction

Zebin You and co-authors wrote a paper from 2025 on a multimodal language model that writes by diffusion rather than one word at a time. Mainstream models predict text left to right, including those that read images. Their model starts from a language diffusion base, which fills in masked text in parallel. They graft a vision encoder and a small connector that maps image features into the language model's space.

Trained on the same instruction data as a left-to-right rival, the model held its own across multimodal tasks. It scaled better as data grew and narrowed the gap to a stronger commercial-grade rival. Against diffusion and hybrid models, it set the best marks in multimodal understanding. Its skill came despite a weaker base on pure text tasks. The team reads that as evidence the diffusion path suits vision-language work and deserves more study.

Catalog record

CREATORZebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, Chongxuan Li, Large Models, Intelligent Governance Engineering Research Center of Next-Generation, Intelligent Search, Recommendation, MOE Ant Group
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16933

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, Chongxuan Li, Large Models, Intelligent Governance Engineering Research Center of Next-Generation, Intelligent Search, Recommendation, MOE Ant Group Abstract In this work, we introduce LLaDA-V , a purely diffusion-based Multimodal Large Language Model (MLLM) that integrates visual instruction tuning with masked diffusion models, representing a departure from the autoregressive paradigms dominant in current multimodal approaches. Built upon LLaDA, a representative large language diffusion model, LLaDA-V incorporates a vision encoder and MLP connector that projects visual features into the language embedding space, enabling effective multimodal alignment. Our empirical investigation reveals several intriguing results: First, LLaDA-V demonstrates promising multimodal performance despite its language model being weaker on purely textual tasks than counterparts like LLaMA3-8B and Qwen2-7B. When trained on the same instruction data, LLaDA-V is highly competitive to LLaMA3-V across multimodal tasks with better data scalability. It also narrows the performance gap to Qwen2-VL, suggesting the effectiveness of its architecture for multimodal tasks. Second, LLaDA-V achieves state-of-the-art performance in multimodal understanding compared to existing hybrid autoregressive-diffusion and purely diffusion-based MLLMs. Our findings suggest that large language diffusion models show promise in multimodal contexts and warrant further investigation in future research. Project page and codes: https://ml-gsai.github.io/LLaDA-V-demo/ . 1 Introduction Multimodal Large Language Models (MLLMs) are capable of processing multiple input modalities—including images [ 1 , 2 , 3 , 4 , 5 , 6 ] , audio [ 7 , 8 , 9 ] , and video [ 10 , 11 , 12 ] —alongside text, and can generate natural language responses that follow given diverse instructions. Despite significant advancements in MLLMs, most existing approaches predominantly rely on autoregressive models [ 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] , leaving substantial room for exploring alternative probabilistic modeling approaches. Recent attempts to incorporate diffusion models [ 22 , 23 , 24 , 25 , 26 ] into MLLMs have predominantly adopted one of two strategies: either leveraging autoregressive models to provide strong language modeling capabilities [ 27 , 28 , 29 , 30 , 31 , 32 ] , or employing discrete diffusion-based approaches with limited language modeling capacity, which consequently leads to suboptimal performance [ 33 , 34 ] . Encouragingly, recent advances in discrete diffusion models [ 25 , 26 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 ] have shown promising potential to overcome these limitations. In particular, LLaDA [ 42 ] has demonstrated performance competitive with LLaMA3-8B-Instruct [ 18 ] through large-scale pre-training and SFT, while retaining favorable scaling properties. Nevertheless, while LLaDA has shown remarkable progress in language modeling, its capabilities and potential in multimodal understanding remain largely unexplored. Therefore, this naturally raises a key research question: Can an purely diffuision based MLLM (both training and sampling) achieve performance compared to autoregressive-based models? In this paper, we explore how to effectively extend large language diffusion models to encompass strong multimodal understanding capabilities, focusing on the visual instruction tuning framework [ 1 ] , which has demonstrated remarkable effectiveness across various autoregressive-based MLLMs. In particular, we introduce a vision encoder (e.g., SigLIP 2 [ 44 ] ) and an MLP connector to map visual features into the LLaDA language embedding space, allowing joint processing of visual and textual inputs. Furthermore, we extend LLaDA’s training objective to handle multi-turn multimodal dialogues, investigate various attention mechanism structures, adapt inference procedures for multimodal conversations, and develop a multi-stage training strategy. These comprehensive investigations result in LLaDA-V , a purely diffusion-based MLLM. (a) LLaDA-V vs. LLaMA3-V (b) LLaDA-V vs. non-autoregressive MLLMs Figure 1 : Benchmark Results. (a) LLaDA-V demonstrates superior performance on more benchmarks compared to LLaMA3-V when trained on the same dataset, particularly excelling in multidisciplinary knowledge and mathematical reasoning tasks. (b) LLaDA-V achieves state-of-the-art performance in multimodal understanding among both hybrid autoregressive-diffusion (such as MetaMorph [ 31 ] and Show-o [ 28 ] ) and purely diffusion-based models. We first compare the data scalability of LLaDA-V to that of LLaMA3-V (our autoregressive baseline with LLaMA3-8B as the language tower) by varying the amount of instruction tuning data. LLaDA-V demonstrates stronger data scalability on several benchmarks, particularly excelling in tasks involving multidisciplinary knowledge and mathematical reasoning (see Fig. 3 ). Furthermore, we benchmark LLaDA-V against autoregressive, hybrid autoregressive-diffusion, and pure diffusion models across 18 diverse multimodal tasks. Notably, when comparing with LLaMA3-V, we observe an interesting and promising phenomenon: despite with a slightly weaker language tower, our model achieves superior performance across 11 tasks (see partial results in Fig. 1 (a) and more details in Section 4 ). Similarly, when compared to the powerful autoregressive Qwen2-VL [ 5 ] , despite LLaDA being considerably weaker than Qwen2-7B, LLaDA-V narrows the performance gap significantly, achieving comparable results on some benchmarks such as MMStar [ 45 ] (60.1 vs. 60.7). Furthermore, our model achieves state-of-the-art performance compared to existing hybrid autoregressive-diffusion models [ 28 , 29 , 30 , 31 , 32 ] and pure diffusion models [ 33 , 34 ] (see Fig. 1 (b)). Collectively, all these findings demonstrate not only the effectiveness of the LLaDA-V framework but also the promise of diffusion models on multimodal understanding. In summary, our key contributions are as follows: • We introduce LLaDA-V, a purely diffusion-based MLLM for multimodal understanding. • We demonstrate that LLaDA-V benefits from data scaling and achieves superior scalability across multiple benchmarks when compared to our autoregressive baseline, LLaMA3-V. • LLaDA-V achieves state-of-the-art results among both hybrid and purely diffusion-based MLLMs. 2 Preliminaries In this section, we briefly introduce large language diffusion models, which serve as the language tower in our work, and visual instruction tuning, which forms the basis of our multimodal framework. Large Language Diffusion Models. Large language models (LLMs) are currently experiencing rapid development. The predominant LLMs [ 13 , 14 , 15 , 16 , 17 , 18 , 19 ] are primarily trained using autoregressive modeling. Unlike autoregressive approaches, discrete diffusion models [ 22 , 25 ] offer an alternative paradigm for language modeling. Masked diffusion models [ 26 , 35 ] , a specific variant of discrete diffusion, have shown impressive results across multiple domains [ 37 , 38 , 39 , 40 , 41 , 42 , 46 , 47 , 48 , 43 ] . Among them, LLaDA [ 42 ] has demonstrated comparable performance with strong AR models like LLaMA3-8B-Instruct [ 18 ] , while maintaining the unique properties of masked diffusion models. Specifically, LLaDA employs a masked diffusion process that differs fundamentally from autoregressive approaches. Formally, let \boldsymbol{x}_{0}=[\boldsymbol{x}^{i}]_{i=1}^{N} represent a sentence comprising N tokens, and let [M] denote a special mask token. LLaDA defines a model distribution p_{\boldsymbol{\theta}}(\boldsymbol{x}_{0}) through a forward and a reverse process. In the forward process, LLaDA first samples a time step t uniformly from the interval [0,1] . Subsequently, each token in \boldsymbol{x}_{0} is replaced by [M] with probability t , yielding the corrupted sentence \boldsymbol{x}_{t} . In the reverse process, LLaDA commences with a sentence composed entirely of [M] tokens and iteratively predicts these masked

Read the complete text