VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Models
Hao Wang, Xiaobao Wei, Jingyang He, Chengyu Bai, Chun-Kai Fan, Jiajun Cao, Jintao Chen, Ying Li, Shanyu Rong, Ming Lu, Xiaozhu Ju, Jian Tang, Shanghang Zhang · 2026
Read VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Models by Hao Wang, Xiaobao Wei, Jingyang He, Chengyu Bai, Chun-Kai Fan, Jiajun Cao, Jintao Chen, Ying Li, Shanyu Rong, Ming Lu, Xiaozhu Ju, Jian Tang, Shanghang Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10485.txt.
Introduction
Hao Wang and twelve colleagues give robot VLA models a better sense of space. Spatial reasoning matters for robot handling. Yet current VLA models pretrain visual backbones on 2D images without explicit 3D supervision. Their 2026 paper proposes VEGA, Visual Encoder Grounding Alignment.
Existing implicit spatial grounding aligns VLA features with 3D-aware foundation models. But it relies on layer search and aligns at the language token level, where spatial structure is already entangled with linguistic semantics. VEGA instead directly aligns the VLA visual encoder's output with spatially aware features from DINOv2-FiT3D. That model is a DINOv2 fine-tuned with multi-view 3D Gaussian Splatting supervision. Alignment uses a lightweight projector trained with cosine similarity loss alongside action prediction. It is discarded at inference, adding no overhead. On simulation and real-world handling tasks, VEGA beat existing implicit spatial grounding baselines, a new state of the art.
Catalog record
| CREATOR | Hao Wang, Xiaobao Wei, Jingyang He, Chengyu Bai, Chun-Kai Fan, Jiajun Cao, Jintao Chen, Ying Li, Shanyu Rong, Ming Lu, Xiaozhu Ju, Jian Tang, Shanghang Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10485
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Models Hao Wang, Xiaobao Wei, Jingyang He, Chengyu Bai, Chun-Kai Fan, Jiajun Cao, Jintao Chen, Ying Li, Shanyu Rong, Ming Lu, Xiaozhu Ju, Jian Tang, Shanghang Zhang Abstract Precise spatial reasoning is fundamental to robotic manipulation, yet the visual backbones of current vision-language-action (VLA) models are predominantly pretrained on 2D image data without explicit 3D geometric supervision, resulting in representations that lack accurate spatial awareness. Existing implicit spatial grounding methods partially address this by aligning VLA features with those of 3D-aware foundation models, but they rely on empirical layer search and perform alignment on LLM-level visual tokens where spatial structure has already been entangled with linguistic semantics, limiting both generalizability and geometric interpretability. We propose VEGA (Visual Encoder Grounding Alignment), a simple yet effective framework that directly aligns the output of the VLA’s visual encoder with spatially-aware features from DINOv2-FiT3D, a DINOv2 model fine-tuned with multi-view consistent 3D Gaussian Splatting supervision. By performing alignment at the visual encoder output level, VEGA grounds spatial awareness before any linguistic entanglement occurs, offering a more interpretable and principled alignment target. The alignment is implemented via a lightweight projector trained with a cosine similarity loss alongside the standard action prediction objective, and is discarded at inference time, introducing no additional computational overhead. Extensive experiments on simulation benchmark and real-world manipulation tasks demonstrate that VEGA consistently outperforms existing implicit spatial grounding baselines, establishing a new state-of-the-art among implicit spatial grounding methods for VLA models. 1 Introduction Robotic manipulation requires precise spatial reasoning about objects, contacts, and intricate geometric relationships in the 3D physical world ( 34 ; 23 ) . Recent vision-language-action (VLA) models ( 21 ; 20 ; 4 ; 3 ; 5 ; 47 ; 2 ) have demonstrated remarkable instruction-following capabilities by leveraging large-scale vision-language pretraining, yet their visual backbones are predominantly trained on unstructured 2D image data without access to any explicit 3D geometric supervision ( 27 ; 9 ; 18 ) . As a consequence, the visual representations learned by these models lack accurate spatial awareness, limiting their ability to reason about object depths, spatial configurations, and viewpoint variations that are essential for precise manipulation in the real world ( 6 ; 17 ) . While such models can succeed on tasks where coarse 2D appearance cues are sufficient, they tend to generalize poorly when manipulation depends on fine-grained spatial understanding, such as estimating object height, inferring relative positions, or adapting to novel viewpoints ( 33 ; 12 ; 43 ) . Figure 1: Comparison of spatial grounding paradigms for VLA models. (1) Explicit Spatial Grounding augments VLA inputs with estimated depth maps from monocular depth estimators (e.g., Depth Anything), introducing additional inference overhead and error propagation. (2) Implicit Spatial Grounding via LLM-level Token Alignment aligns visual features at the LLM token level, where spatial structure is entangled with linguistic semantics. (3) Ours (VEGA) directly aligns visual encoder outputs with spatially-aware features via a lightweight projector, grounding spatial awareness before any linguistic entanglement occurs. To bridge this gap, existing approaches generally fall into two categories. The first category explicitly incorporates 3D information into VLA models, either by augmenting inputs with depth maps or point clouds from additional sensors 32 ; 36 ; 1 , or by estimating 3D structure from 2D images via depth estimators 40 ; 41 . Additionally, some approaches directly supervise the VLA’s geometric reasoning by fine-tuning on curated robotic datasets enriched with human-annotated 3D bounding boxes, trajectory keypoints, or spatial coordinates ( 46 ; 15 ) . While effective, these methods face practical limitations: depth sensors introduce hardware and calibration overhead, depth maps are often noisy for transparent or reflective objects ( 22 ) , and large-scale 3D annotated datasets remain scarce and costly to collect ( 30 ; 11 ) . The second category pursues a more hardware-agnostic alternative via implicit spatial grounding, incorporating features from frozen 3D foundation models ( 39 ) into VLA representations through representation alignment supervision ( 25 ; 35 ; 13 ) or feature fusion ( 26 ) . This direction is appealing precisely because it requires no additional sensors or 3D annotations at inference time. However, these methods share two fundamental limitations. First, they rely on empirical layer search to identify which intermediate layer to align, making the alignment target sensitive to hyperparameter choice and difficult to generalize. Second, alignment is performed on LLM-level visual tokens where visual features have already been fused with linguistic context, conflating geometric structure with semantic associations and losing geometric interpretability. To address this limitation, we propose VEGA ( V isual E ncoder G rounding A lignment), a simple yet effective representation alignment framework that directly aligns VLA visual encoder outputs with spatially-aware features from a 3D-aware foundation model. Rather than relying on empirical layer search or aligning LLM-level tokens, VEGA supervises the visual encoder outputs of the VLA backbone with features extracted from FiT3D ( 44 ) , a DINOv2 model fine-tuned with 3D-aware supervision via multi-view consistent 3D Gaussian Splatting ( 19 ) . By performing alignment directly at the visual encoder level, VEGA ensures that spatial awareness is grounded at the representation stage before linguistic entanglement occurs, offering a more geometrically interpretable alignment target. This design introduces no additional computational overhead at inference time. Extensive experiments on the RoboTwin 2.0 benchmark ( 8 ) and real-world robotic manipulation tasks demonstrate that VEGA consistently outperforms existing methods, establishing a new state-of-the-art among implicit spatial grounding methods for VLA models. In summary, our main contributions are as follows: • We identify that existing implicit spatial grounding methods suffer from two compounding issues: reliance on empirical layer search, and performing alignment on LLM-level tokens where geometric structure has been conflated with linguistic semantics. We show that directly aligning at the visual encoder output level resolves both issues simultaneously. • We propose VEGA, a simple yet effective alignment framework that supervises VLA visual encoder outputs with DINOv2-FiT3D features via a lightweight projector, injecting 3D spatial awareness at the most geometrically meaningful representation stage without any additional inference overhead. • Extensive experiments on RoboTwin and real-world manipulation tasks demonstrate that VEGA consistently outperforms existing methods, achieving state-of-the-art performance among implicit spatial grounding methods for VLA models. 2 Related Work 2.1 Vision-Language-Action Models The integration of Large Language Models (LLMs) with robotic control has led to the emergence of Vision-Language-Action (VLA) models, which have significantly advanced the field of embodied AI. Early pioneering works, such as RT-1 ( 5 ) and RT-2 ( 47 ) , demonstrated that pre-trained Vision-Language Models (VLMs) could be effectively repurposed to map raw visual observations and natural language instructions directly into low-level robot actions. Following this paradigm, open-source models like OpenVLA ( 21 ) and Octo ( 37 ) have become standard baselines, leveraging powerful 2D vision backbones (e.g., SigLIP ( 45 ) , DINOv2 ( 29 ) ) and large-scale cross-embodiment datasets ( 30 ) to achieve remarkable zero-shot generalization. More recently, architectures such as \pi_{0} ( 4 ; 3 ) and GR00T ( 2 ) have introduced flow-matching and continuous action generation, further improving dexterous manipulation capabilities. Despite their semantic reasoning capabilities, the visual backbones of these mainstream VLAs are primarily