Avi: A 3D Vision-Language Action Model Architecture generating Action from Volumetric Inference
Harris Song, Long Le · 2025
Read Avi: A 3D Vision-Language Action Model Architecture generating Action from Volumetric Inference by Harris Song, Long Le (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21746.txt.
Introduction
Harris Song and Long Le reframe robot control as three-dimensional reasoning instead of learned reflexes. Most vision-language-action models read flat images. They are trained end to end on action demonstrations. That ties them to their training tasks and viewpoints. It also hides their decisions inside opaque policies.
Avi, their architecture, works on 3D point clouds. A multimodal language model predicts the next point cloud of the scene. The robot's action is then computed from that prediction with classical geometric transformations. There is no training on action tokens at all. Because the reasoning lives in 3D, the behavior should survive occlusions and moved cameras. New viewpoints that break flat-image policies should also be handled. Early results support 3D vision-language reasoning as a scalable, sturdier foundation for general robots.
Catalog record
| CREATOR | Harris Song, Long Le |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21746
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Avi: A 3D Vision-Language Action Model Architecture generating Action from Volumetric Inference Harris Song, Long Le Abstract We propose Avi , a novel 3D Vision-Language-Action (VLA) architecture that reframes robotic action generation as a problem of 3D perception and spatial reasoning, rather than low-level policy learning. While existing VLA models primarily operate on 2D visual inputs and are trained end-to-end on task-specific action policies, Avi leverages 3D point clouds and language-grounded scene understanding to compute actions through classical geometric transformations. Most notably, Avi does not train on previous action tokens, rather, we build upon a 3D Multi-modal Large Language Model (MLLM) to generate the next point cloud and explicitly calculate the actions through classical transformations. This approach enables generalizable behaviors that are robust to occlusions, camera pose variations, and changes in viewpoint. By treating the robotic decision-making process as a structured reasoning task over 3D representations, Avi bridges the gap between high-level language instructions and low-level actuation without requiring opaque policy learning. Our preliminary results highlight the potential of 3D vision-language reasoning as a foundation for scalable, robust robotic systems. Check it out at avi-3drobot.github.io . 1 Introduction Figure 1: The left image represents the starting position of the scene. The green voxels represent the predicted next time stamp. The right image represents the end time stamp, and the series of images in between indicate the rollout. Vision-Language-Action (VLA) models have recently gained significant attention in the robotics and machine learning communities Black et al. ; Team et al. 2025 . While these models have demonstrated impressive capabilities in connecting high-level natural language instructions with actionable robot policies, the vast majority of current VLAs operate solely on 2D image inputs Chi et al. 2023 ; Li et al. 2025b ; Black et al. . This reliance on 2D perception imposes fundamental limitations: reasoning about depth, object geometry, and fine-grained spatial relations becomes indirect and often error-prone. There are other novel Vision-Language Action models that infer based on other sources, such as implicitly 3D Point Clouds Yang et al. 2025 ; Hou et al. 2025 , but they treat the robotics problem as an end-to-end action generation problem and implicitly assume that the model is generating action tokens in a scene and robot-specific setup, decreasing the reproducibility of these results. In this work, we propose to move beyond both the 2D perception and action-based paradigm by training a VLA model that natively operates on 3D representations, specifically point clouds. Our approach is built upon ShapeLLM-Omni, a 3D Multi-Modal Language Model (3D MMLM), which we finetune to condition on both natural language commands and a 3D point cloud of the scene. Ye et al. 2025 Rather than directly outputting low-level joint actions, the model predicts a delta point cloud that represents the desired post-condition of the manipulated object(s). Robot joint actions are then derived through traditional inverse kinematics, aligning the end-effector to the predicted “after” state of the point cloud. In summary, we present two main contributions: 1. AVI (Action from Volumetric Inference) : a novel architecture that integrates a 3D Multi-Modal Language Model to infer actions through volumetric reasoning, rather than directly generating action tokens. More importantly, our architecture doesn’t require training on previous action tokens, but rather, only previous depth maps . This approach shifts the focus from language-to-action to language-to-geometry, enabling richer spatial grounding. 2. Location Quantization for 3D MLLMs : a general technique for discretizing spatial information that allows pretrained 3D MLLMs to generalize at the object level rather than at the scene level. Current state-of-the-art 3D MLLMs, specifically ShapeLLM-Omni Ye et al. 2025 , are built on training assets with online 3D Models rather than entire scene generation. Developing a simple location quantization technique helps us overcome this technical barrier while demonstrating the effectiveness of our architecture. 2 Related Work Our work lies at the intersection of two separate trends. The first one is in robotics, where the increase in large robotics datasets such as Droid, Open X Embodiment has led to the development of many novel architectures Khazatsky et al. 2024 ; Collaboration et al. 2023 . Figure 2: Comparison against related work. (1) describes Shape LLM Omni. (2) describes our work, Avi. (3) describes the Unified Video Action Model. (4) describes Robot 4D Generation. (5) describes 3D Foundation Policy, FP3. The second trend is within computer vision, where there is now a strong emphasis towards understanding the 3D Space. 3D Reconstruction techniques like NeRF and Gaussian Splatting are capable of learning camera parameters for 3D spatial representation. In addition, image encoders are capable of deriving semantic information, and nowadays, are capable of deriving enough semantic information for reconstruction in the 3D Space. 2.1 Understanding the Robotics Policy Trend There are two additional trends in the robotics space to analyze. The first trend is that most robotics architectures are based on training action tokens. The second robotics-based trend involves injecting signals into visual components, such as through an image encoder that can extract semantic information. Action Generation We usually dictate these problems as a A_{t-n}...A_{t-1}\rightarrow A_{t} Zhao et al. 2023 ; Torabi et al. 2018 . These prior Vision-Language-Action (VLA) methods directly predict robot-specific action tokens, trained to mimic collected demonstrations. For example, PerAct Shridhar et al. 2022 , CLIPort Shridhar et al. 2021 , and Transporter Networks Zeng et al. 2021 couple visual perception with learned policies tailored to specific robot morphologies. Similarly, Unified Video Action Black et al. and related works integrate video-based action representations, but remain dependent on large-scale, robot-specific data. 2D Image Pretraining A significant line of research leverages 2D vision pretraining for robotics. We treat this as a \{A_{t-n},I_{t-n}\}...\{A_{t-1},I_{t-1}\}\rightarrow A_{t} Yen-Chen et al. Yen-Chen et al. 2020 demonstrated transfer learning in manipulation policies by reusing COCO-pretrained backbones. Radosavovic et al. Radosavovic et al. 2023 applied masked autoencoding to pretrain ViTs on large-scale 2D data before transferring to robotics. More recent work has focused on scaling such paradigms: Lift3D Jia et al. 2024 extends 2D representations into implicit depth and point-cloud embeddings, while 3D-MVP Qian et al. 2024 and FVP Hou et al. 2025 extend multiview pretraining to robotic manipulation and 4D video, respectively. World-consistent diffusion models Zhang et al. 2025 also highlight the utility of 2D/temporal video modeling as a precursor to structured 3D reasoning. While effective, these approaches remain fundamentally limited by their reliance on 2D visual input for geometry. More notably, there are a small subset of papers, such as the Unified Video Action Model, that generate \{A_{t},I_{t}\} Li et al. 2025b . Their use of image generation through a diffusion policy is relatively novel, although their policy still depends on a shared latent space with an action diffusion. 2.2 Understanding the Trend of 3D Computer Vision There are many papers in the 3D Computer Vision space. We discuss a few from the conversion of 2D to 3D space, and then discuss new papers discussing 3D Large Language Models that first use arbitrary 3D environments, such as game assets for their dataset, then shift to using the real-world 3D world models as 3D spaces become more wide-spread. Generating 3D Environments InstantSplat introduces a method to generate implicit 3D representations of a scene without explicit camera parameters, and is one