vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models

Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu · 2026

Read vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models by Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02204.txt.

Introduction

Peiqi Yin and colleagues present vLLM-Omni, a serving system built for any-to-any multimodal models handling text, images, video, and audio. Such models combine autoregressive language models, diffusion transformers, and other specialized parts. Existing serving systems target a single paradigm and force developers to hand-wire cross-stage interactions at steep performance cost. The new system decomposes architectures into stages linked as a graph, each served independently with per-stage batching and flexible processor allocation. Experiments show job completion time falling by up to nine tenths against baselines.

Catalog record

CREATORPeiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35399
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.02204

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu Abstract Any-to-any multimodal models that jointly handle text, images, video, and audio represent a significant advance in multimodal AI. However, their complex architectures (typically combining multiple autoregressive LLMs, diffusion transformers, and other specialized components) pose substantial challenges for efficient model serving. Existing serving systems are mainly tailored to a single paradigm, such as autoregressive LLMs for text generation or diffusion transformers for visual generation. They lack support for any-to-any pipelines that involve multiple interconnected model components. As a result, developers must manually handle cross-stage interactions, leading to huge performance degradation. We present vLLM-Omni, a fully disaggregated serving system for any-to-any models. vLLM-Omni features a novel stage abstraction that enables users to decompose complex any-to-any architectures into interconnected stages represented as a graph, and a disaggregated stage execution backend that optimizes resource utilization and throughput across stages. Each stage is independently served by an LLM or diffusion engine with per-stage request batching, flexible GPU allocation, and unified inter-stage connectors for data routing. Experimental results demonstrate that vLLM-Omni reduces job completion time (JCT) by up to 91.4% compared to baseline methods. The code is public available at https://github.com/vllm-project/vllm-omni . † † affiliationnotice: * Core Contributor. 1 Introduction Traditional large language models (LLMs) ( 1 ; 5 ; 67 ) have achieved remarkable performance in language understanding and reasoning tasks, such as question answering ( 35 ; 7 ) , summarization ( 73 ; 75 ) , and code generation ( 21 ) . However, LLMs are constrained to text-only modalities, especially for the output. The growing demand for processing multimodal data has motivated the development of multimodal models that extend LLMs with encoders and decoders for diverse modalities ( 63 ; 10 ; 69 ; 42 ) . A key trend is the emergence of any-to-any multimodal models ( 65 ; 66 ; 42 ; 60 ) , unified architectures that seamlessly understand and generate across text, images, video, and audio through end-to-end training (Figure 1 ). This unification enables more flexible cross-modal reasoning and interaction compared to separate understanding and generation pipelines. Recent any-to-any models have achieved SOTA performance across multimodal understanding tasks such as image captioning ( 64 ; 26 ) , visual question answering ( 51 ; 50 ) , and voice-based assistance ( 14 ; 38 ) , as well as generation tasks including image editing ( 41 ; 33 ) , speech translation ( 11 ; 12 ) , and text-to-speech synthesis ( 13 ; 61 ) . Figure 1 : Any-to-any multimodal model architecture. The emergence of these any-to-any patterns leads to more complex model structures than those of traditional LLMs. For example, to support audio generation, modern any-to-any models such as Qwen-Omni ( 65 ; 66 ) adopt a Thinker–Talker architecture that connects two autoregressive (AR) LLMs, one dedicated to generating text tokens and the other to generating audio tokens. To support image generation, models like GLM-Image ( 69 ) typically use an AR LLM to understand the input and then connect it to a diffusion transformer (DiT) ( 46 ) for visual synthesis. More advanced any-to-any models further integrate multiple AR and DiT components to simultaneously support both audio and visual outputs within a unified pipeline ( 42 ; 25 ) . Such complexity in model structure introduces substantial challenges for efficient model serving. Existing serving frameworks are typically specialized for a single generation paradigm. LLM serving frameworks such as vLLM ( 36 ) and SGLang ( 74 ) are optimized for autoregressive LLM decoding and designed for text-only generation, while diffusion serving frameworks ( 57 ; 22 ) are optimized for DiT denoising and designed for image and video generation. As a result, these frameworks lack native support for any-to-any pipelines that involve multiple autoregressive LLMs, DiT models, or other specialized neural components that must interact in customized ways to produce multi-modal outputs. Developers are thus unable to leverage these frameworks and instead resort to custom implementations that are tightly coupled with specific models, resulting in poor performance and limited extensibility. To address the challenges, we propose vLLM-Omni, a fully disaggregated serving system for any-to-any models, featuring a stage abstraction frontend and stage execution backend. Unlike existing LLM serving frameworks that operate on a single AR decoding or DiT denoising stage, vLLM-Omni supports complex any-to-any architectures by introducing the concept of a stage graph . Through vLLM-Omni’s stage abstraction, users can decompose an any-to-any model into multiple stages and explicitly define the pipeline as a stage graph. The nodes represent model stages (e.g., AR or DiT), and edges correspond to user-defined functions that transform and route intermediate data to subsequent stages. Given this stage graph specification, vLLM-Omni delivers efficient serving for any-to-any models by leveraging a disaggregated stage-execution backend that optimizes execution and resource utilization across stages. Each stage is independently served by a specialized execution engine, in which vLLM for LLM stages and a dedicated diffusion engine for DiT stages. Each engine performs per-stage request batching to maximize resource utilization. Users can flexibly allocate computing accelerators and memory resources to each stage according to its computational characteristics. To support the disaggregated execution across the entire pipeline, vLLM-Omni employs a unified connector between stages for flexible and customized intermediate data transfer. Our evaluation shows that vLLM-Omni achieves substantial performance improvements across diverse multimodal models. For Qwen3-Omni, vLLM-Omni reduces job completion time by up to 91.4% compared to baseline method. To summarize, we make the following contributions. \bullet We introduce vLLM-Omni, a fully disaggregated serving system for any-to-any multimodal models. vLLM-Omni propose a stage graph abstraction, enabling native support for multi-stage model pipelines. \bullet We present a stage execution backend that enables stage-wise optimizations, a unified connector for data transfer, and a diffusion engine for visual generation. \bullet Our experiments demonstrate that vLLM-Omni consistently outperforms baselines across diverse any-to-any models and tasks. 2 Background & Motivation 2.1 Any-to-Any Multimodal Models Any-to-any multimodal models ( 66 ; 65 ; 25 ; 42 ; 60 ) extend the capabilities of text-only LLMs by processing and generating outputs across diverse modalities, including text, image, video, and audio. Unlike standard LLMs, Any-to-any models employ specialized architectures for cross-modality understanding and generation. For multimodal comprehension, modern models adopt dedicated encoders (audio encoders such as Whisper ( 49 ) and Audio Transformer; vision encoders such as ViT ( 6 ) and SigLIP ( 55 ) ) that map multimodal inputs into a shared embedding space connected to an LLM backbone. For multimodal generation, the LLM backbone generate embedding outputs and parsed to modality-specific decoders, including text-to-speech models ( 31 ; 19 ) and image/video generation models ( 20 ; 37 ) , to produce diverse output modalities. 1 1 1 We refer the model with multiple input and output modalities as “any-to-any” models in this work. Noted that not all any-to-any models support all input and output modalities. Some models may support multimodal inputs (e.g., text, image, audio) but only generate text and images, or text and audio. The combination

Read the complete text