3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models
Changti Wu Hang Yuan Bailing Wang Cong Huang Kai Chen · 2026
Read 3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models by Changti Wu Hang Yuan Bailing Wang Cong Huang Kai Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24393.txt.
Introduction
Yu and co-authors present 3D-Mix, a plug-in module that adds 3D information to robot control models. Models trained mostly on 2D data often lack the spatial sense that manipulation needs. The team compares nine ways to fuse VGGT 3D features, and gated fusion guided by task meaning wins. 3D-Mix drops into GR00T-style and pi-style systems without changing their core parts. Across nine model variants, it adds an average gain of seven percent on the out-of-domain SIMPLER benchmark.
Catalog record
| CREATOR | Changti Wu Hang Yuan Bailing Wang Cong Huang Kai Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37619 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.24393
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models Changti Wu Hang Yuan Bailing Wang Cong Huang Kai Chen Abstract Vision-Language-Action (VLA) models leverage Multimodal Large Language Models (MLLMs) for robotic control, but recent studies reveal that MLLMs exhibit limited spatial intelligence due to training predominantly on 2D data, resulting in inadequate 3D perception for manipulation tasks. While recent approaches incorporate specialized 3D vision models such as VGGT to enhance spatial understanding, they employ diverse integration mechanisms without systematic investigation, leaving the optimal fusion strategy unclear. We conduct a comprehensive pilot study comparing nine VGGT integration schemes on standardized benchmarks and find that semantic-conditioned gated fusion , which adaptively balances 2D semantic and 3D geometric features based on task context, achieved the strongest performance among all nine evaluated fusion schemes in our pilot study. We present 3D-Mix , a plug-and-play module that integrates into diverse VLA architectures (GR00T-style and π \pi -style) without modifying existing MLLM or action expert components. Experiments across six MLLM series (nine model variants, 2B–8B parameters) on SIMPLER and LIBERO show that 3D-Mix delivers consistent performance gains, averaging + 7.0 % +7.0\% on the out-of-domain (OOD) SIMPLER benchmark across all nine GR00T-style variants, establishing a principled approach for enhancing spatial intelligence in VLA systems. https://github.com/ZGC-EmbodyAI/3DMix-for-VLA 1 Introduction Vision-Language-Action (VLA) models have emerged as a promising paradigm for robotic manipulation, unifying visual perception, language understanding, and action generation within end-to-end learned systems. By leveraging Multimodal Large Language Models (MLLMs) as semantic encoders coupled with specialized action prediction modules, VLA models enable robots to follow natural language instructions and generalize across diverse tasks. Despite their success, current VLA models face a fundamental limitation: MLLMs are predominantly pre-trained on 2D image-text corpora lacking explicit 3D geometric supervision. As a result, they exhibit limited depth perception and struggle with precise spatial reasoning, capabilities critical for manipulation tasks that require accurate localization, grasp pose estimation, and spatial relationship understanding. To address this limitation, recent research has explored integrating specialized 3D vision models into VLA architectures. Among various 3D encoders, VGGT (Visual Geometry Grounded Transformer) ( 19 ) has emerged as a compelling choice due to its ability to extract rich geometric features from single or multi-view images in a unified framework. However, existing approaches to incorporating VGGT into VLA systems employ diverse fusion strategies without systematic comparison, leaving the optimal integration strategy unclear. This raises several questions: Where should 3D features be injected? How should geometric and semantic information be combined? What fusion mechanisms enable effective cross-modal interaction? We address these questions through a comprehensive empirical study of VGGT integration strategies. We systematically evaluate nine distinct fusion schemes spanning the entire VLA pipeline and identify the key factors that determine fusion effectiveness. We then propose 3D-Mix , a plug-and-play module that integrates VGGT-derived 3D features through semantic-conditioned adaptive gating. Unlike fixed fusion strategies, 3D-Mix employs learnable gates that dynamically balance semantic and geometric information at each spatial location, enabling context-aware fusion without modifying MLLM or action expert internals. Experiments across six MLLM series (nine model variants, 2B–8B parameters; Qwen2.5-VL, Qwen3-VL, RoboBrain2/2.5, MimoEmbodied, RynnBrain) on SimplerEnv and LIBERO show that 3D-Mix consistently improves performance across architectures and model scales. Our contributions are as follows: • We present the first systematic study of VGGT integration schemes for VLA models, evaluating nine fusion strategies to understand how 3D information contributes to manipulation performance. • We propose 3D-Mix , a lightweight plug-and-play module that performs principled 3D-semantic fusion through adaptive gating, achieving consistent improvements without architecture-specific modifications. • We demonstrate 3D-Mix ’s generalizability across multiple VLA architectures and MLLM backbones, providing practical guidelines for enhancing spatial intelligence in VLA systems. 2 Related Work Vision-Language-Action Models. Vision-Language-Action (VLA) models unify perception, reasoning, and control within end-to-end frameworks by leveraging pre-trained MLLMs for generalizable robot policies. Recent works such as OpenVLA ( 7 ) , \pi_{0.5} ( 3 ; 6 ) , and GR00T-N1.6 ( 18 ) have demonstrated effective robotic control by scaling MLLMs with robotic data under the VLA architecture. The VLA architectures used in this paper build upon these works. 3D Fusion for VLA Models. While VLAs are built upon pretrained MLLMs, recent works ( 1 ; 20 ; 21 ) have revealed the limited spatial reasoning capabilities of MLLMs, attributing this primarily to the inability to explicitly model 3D structure. VGGT ( 19 ) has attracted attention as a pretrained Transformer applicable to diverse 3D downstream tasks. Several works ( 8 ; 10 ; 22 ) have accordingly introduced 3D information from VGGT into VLA systems to improve robotic control. However, these works adopt diverse fusion strategies without systematic comparison, leaving the community without principled guidance on how to best incorporate 3D information. We address this gap with a comprehensive pilot study of nine fusion schemes and distill the empirical insights into 3D-Mix , a lightweight plug-and-play module. 3 Pilot Study: Comparative Analysis of VGGT Fusion Schemes To systematically investigate the optimal approach for integrating VGGT-derived 3D geometric information into VLA models, we compare nine distinct fusion schemes in a controlled pilot study. This study identifies the most effective architectural pattern before scaling to full experiments. We first describe the base VLA architecture, then detail each fusion scheme, and finally present the experimental setup. Figure 1: Overview of nine VGGT fusion schemes evaluated in our pilot study. Each scheme integrates VGGT-derived 3D geometric features into the VLA pipeline at a different location or via a different mechanism, ranging from early token-level injection to training-time alignment and cross-attention fusion. 3.1 Base VLA Architecture We adopt a GR00T-style VLA framework comprising an MLLM backbone (Qwen3-VL ( 2 ) ) that encodes multi-view images and language instructions into hidden states \mathbf{H}\in\mathbb{R}^{B\times L\times D} , and a DiT-based flow-matching action expert ( 14 ) that predicts future actions through a flow-matching objective. This modular architecture enables systematic evaluation of different 3D fusion mechanisms without disrupting other components. Detailed architecture specifications are provided in Appendix A.1 . 3.2 Multiple VGGT Fusion Schemes We investigate nine distinct fusion schemes for integrating VGGT-derived 3D geometric features into the base VLA architecture, as illustrated in Figure 1 . Each scheme differs in where and how VGGT features are injected, spanning from early input-level fusion to late action-stage fusion: (1) AE-Fusion : Augments the action expert with dual cross-attention to simultaneously attend to MLLM semantic features and VGGT geometric features during denoising. (2) Early Fusion : Injects VGGT tokens directly into the MLLM input sequence, enabling implicit semantic-geometric interaction via self-attention. (3) Concat Fusion & (4) CrossAttn Fusion : Apply GateMixer preprocessing to VGGT features, then either concatenate with MLLM outputs (Concat) or apply explicit cross-attention before concatenation (CrossAttn). (5) Gated Fusion : Employs learnable gates to dynamically balance semantic and geometric features at each token position based on global semantic context. (6) 3D-Tokens : Appends a special token \langle|\text{vggt}|\rangle supervised to encode 3D geometry via alignment loss; VGGT not required at inference. (7) Middle Layer Injection : Injects VGGT features into an intermediate MLLM layer via adapter-style cross-attention, enabling deeper semantic-geometric integration. (8) Spatial Forcing ( 8 ) : Supervises intermediate MLLM layers with alignment loss during training; discards VGGT at inference with zero overhead. (9)