Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model

John Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim, Jinwoo Shin · 2025

Read Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model by John Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim, Jinwoo Shin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27607.txt.

Introduction

John Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim, and Jinwoo Shin present DUST in 2025, a robot model that joins seeing, language, and action with a world model. A world model predicts how a scene will change. Joining that skill with action choice is hard, since sight data and move data differ a lot in form.

DUST keeps two streams, one for sight and one for action, inside one shared build. Each stream gets its own noise, and the loss keeps their causal links clear. Sampling at test time can then refine moves and views at distinct paces. Gains reach six points on sim tests and ten points on a real robot arm. Transfer to new tasks also improves.

Catalog record

CREATORJohn Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim, Jinwoo Shin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-27607

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model John Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim, Jinwoo Shin Abstract Recently, augmenting vision-language-action models (VLAs) with world-models has shown promise in robotic policy learning. However, it remains challenging to jointly predict next-state observations and action sequences because of the inherent difference between the two modalities. To address this, we propose du al- st ream diffusion (DUST), a world-model augmented VLA framework that handles the modality conflict and enhances the performance of VLAs across diverse tasks. Specifically, we propose a multimodal diffusion transformer architecture that explicitly maintains separate modality streams while enabling cross-modal knowledge sharing. In addition, we propose training techniques such as independent noise perturbations for each modality and a decoupled flow matching loss, which enables the model to learn the joint distribution in a bidirectional manner while avoiding the need for a unified latent space. Furthermore, based on the decoupled training framework, we introduce a sampling method where we sample action and vision tokens asynchronously at different rates, which shows improvement through inference-time scaling. Through experiments on simulated benchmarks such as RoboCasa and GR-1, DUST achieves up to 6% gains over a standard VLA baseline and implicit world-modeling methods, with our inference-time scaling approach providing an additional 2-5% gain on success rate. On real-world tasks with the Franka Research 3, DUST outperforms baselines in success rate by 13%, confirming its effectiveness beyond simulation. Lastly, we demonstrate the effectiveness of DUST in large-scale pretraining with action-free videos from BridgeV2, where DUST leads to significant gain when transferred to the RoboCasa benchmark. 1 Introduction Vision-language-action models (VLAs) have recently emerged as promising approaches for learning general-purpose robotic policies ( Black et al. 2025 ; NVIDIA et al. 2025 ; Brohan et al. 2023 ; Li et al. 2023b ; Kim et al. 2024 ; Luo et al. 2025 ; Shukor et al. 2025 ) . Specifically, VLAs are built upon vision-language models (VLMs), which are pretrained on internet-scale multimodal datasets and excel in visual and textual understanding. Then, VLAs leverage VLM features to generate actions, through action experts ( e.g. , diffusion policy ( Chi et al. 2023 ) ), that generate robotic actions given the current observation and text instruction. As such, VLAs are capable of generating precise actions that generalize to novel objects, scenes, and instructions ( Zawalski et al. 2024 ) . However, despite strong perceptual grounding and instruction following capabilities, VLAs often fail to model how actions affect the environment, and fall short in terms of explicit understanding of underlying physical processes ( Guo et al. 2024 ) . To address this, recent works have augmented VLAs with world-modeling objectives, which train models to predict future visual observations together with actions ( Guo et al. 2024 ; Zheng et al. 2025 ; Liang et al. 2025 ) . Through learning the joint distribution of the two modalities, it enables the models to effectively capture the latent dynamics that govern both actions and their visual results, improving performance and generalization. Previous works utilized unified joint diffusion model structures ( e.g. , see Figure 1(a) ), where the two modalities are concatenated together and modeled with a single unified model ( Guo et al. 2024 ; Huang et al. 2025 ) . However, their design relies on the implicit assumption of the existence of a shared latent space between the modalities. As a result, the model often suffers from mismatch between modalities, where action predictions require low-dimensional, temporally smooth outputs, while future visual observations require high-dimensional, spatially structured outputs. Other approaches adopt a causal design ( e.g. , see Figure 1(b) ), which separates the modalities into distinct models with uni-directional conditioning ( Liang et al. 2025 ; Hu et al. 2025 ) . While this design can handle modality-specific structure, the design inherently limits information flow to a single direction, and prevents bidirectional knowledge transfer. As such, designing world-models and action prediction models remains a challenge due to the trade-off between cross-modal integration and modality-specific fidelity. Contribution. To bridge these contrasting approaches, we propose dual-stream diffusion ( DUST ) , a VLA architecture that preserves distinct modality streams while facilitating information exchange across them (see Figure 1(c) ). DUST employs a multimodal diffusion transformer (MMDiT) ( Esser et al. 2024 ) that maintains separate token streams for actions and future visual observations, each with its own timestep embedding and normalization. The two modality streams interact through shared cross-modal attention layers, allowing bidirectional information flow without collapsing modalities into a single latent space. On top of this architecture, we introduce a decoupled diffusion training algorithm that applies independent noising schedules to each modality, enabling the model to learn causal relationships between them under various noise configurations ( Chen et al. 2025 ; Rojas et al. 2025 ) . The network is optimized via modality-specific flow matching losses, allowing actions and observations to evolve according to their respective statistical structures. Finally, we introduce a unique sampling strategy for DUST that jointly samples action and visual observations. Specifically, in order to handle the difference between the modalities, we introduce asynchronous denoising, where we take diffusion steps on the high-dimensional vision tokens more frequently than the low-dimensional action tokens. As a result, our approach yields scalable test-time scaling that effectively balances efficiency and accuracy. We evaluate the effectiveness and scalability of DUST through extensive evaluations on simulated, real-world, and transfer learning scenarios. To analyze DUST’s pretraining performance, we freeze the backbone VLM ( Li et al. 2025b ) and train the diffusion-based action expert ( Chi et al. 2023 ) across all experiments. In simulation benchmarks, DUST outperforms baselines such as standard VLAs ( e.g. , GR00T-N1.5 ( NVIDIA et al. 2025 ) ) and implicit world-modeling approaches ( e.g. , FLARE ( Zheng et al. 2025 ) ) on the RoboCasa and GR-1 benchmarks, achieving success rate gains of 5% and 6%, respectively. When evaluating on a real Franka Research 3 arm, DUST achieves the highest success rates across diverse pick-and-place tasks, outperforming baselines by over 12%, and demonstrates robust real-world performance and physically consistent predictions across environments. Lastly, we leverage DUST in pretraining with action-free videos ( e.g. , BridgeV2 ( Walke et al. 2023 ) ), and show that DUST exhibits significant gains when transferred to downstream tasks, such as the RoboCasa benchmark. The asynchronous joint diffusion sampling strategy also proves effective at test-time, providing an additional 2–6% boost over naive sampling approaches. (a) Unified Joint Diffusion Model (b) Causal Diffusion Model (c) Dual-Stream Diffusion Model Figure 1: Architectures of world-model augmented VLAs. (a) Unified Joint Diffusion concatenates action and vision tokens and generates both with a single model. (b) Causal Diffusion uses separate models with one-way conditioning. (c) Dual-Stream Diffusion (ours) maintains separate streams for each modality while enabling cross-modal knowledge transfer through shared attention. 2 Related Works Vision-language-action models (VLAs). VLAs have recently emerged as a promising paradigm for general-purpose robot policy learning, extending vision–language models (VLMs) pretrained on internet-scale multimodal datasets. Building on the strong representational capacity of VLMs ( Dai et al. 2023 ; Team 2024 ; Xiao et al. 2024 ) , VLA architectures adapt them for robotics by either generating actions autoregressively ( Kim et

Read the complete text