OG-VLA: 3D-Aware Vision Language Action Model via Orthographic Image Generation
Ishika Singh, Ankit Goyal, Stan Birchfield, Dieter Fox, Animesh Garg, Valts Blukis · 2025
Read OG-VLA: 3D-Aware Vision Language Action Model via Orthographic Image Generation by Ishika Singh, Ankit Goyal, Stan Birchfield, Dieter Fox, Animesh Garg, Valts Blukis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01196.txt.
Introduction
Robots that take spoken orders still fumble when the camera angle or the robot's pose shifts. Ishika Singh and five co-authors present OG-VLA in 2025, a model that joins language-guided action with a firm sense of three-dimensional space. The goal is steady, exact moves even in scenes the robot has not seen before.
The method turns camera views and depth data into a point cloud, then renders that cloud from fixed top-down style views. Because input and output share the same fixed views, pose changes matter less. The team builds on large vision and language models to keep the broad skill of those systems while adding the care that fine hand work needs.
Catalog record
| CREATOR | Ishika Singh, Ankit Goyal, Stan Birchfield, Dieter Fox, Animesh Garg, Valts Blukis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01196
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OG-VLA: 3D-Aware Vision Language Action Model via Orthographic Image Generation Ishika Singh, Ankit Goyal, Stan Birchfield, Dieter Fox, Animesh Garg, Valts Blukis Abstract We introduce OG-VLA , a novel architecture and learning framework that combines the generalization strengths of Vision Language Action models (VLAs) with the robustness of 3D-aware policies. We address the challenge of mapping natural language instructions and multi-view RGBD observations to quasi-static robot actions. 3D-aware robot policies achieve state-of-the-art performance on precise robot manipulation tasks, but struggle with generalization to unseen instructions, scenes, and objects. On the other hand, VLAs excel at generalizing across instructions and scenes, but can be sensitive to camera and robot pose variations. We leverage prior knowledge embedded in language and vision foundation models to improve generalization of 3D-aware keyframe policies. OG-VLA projects input observations from diverse views into a point cloud which is then rendered from canonical orthographic views, ensuring input view invariance and consistency between input and output spaces. These canonical views are processed with a vision backbone, a Large Language Model (LLM), and an image diffusion model to generate images that encode the next position and orientation of the end-effector on the input scene. Evaluations on the Arnold and Colosseum benchmarks demonstrate state-of-the-art generalization to unseen environments, with over 40% relative improvements while maintaining robust performance in seen settings. We also show real-world adaption in 3 to 5 demonstrations along with strong generalization. Videos and resources at https://og-vla.github.io 1 Introduction Figure 1: OG-VLA illustration. OG-VLA represents robot end-effector keyframes with easy-to-decode annotations on orthographic images in a set of canonical views. This output encoding enables action prediction via image generation, and using canonical views achieves invariance to input camera poses. The red hotspot is the predicted end-effector position in each image. In this example, the hotspot is indicating the 3D point for approaching the drawer handle to open it. The yellow, blue and green hotspots work in tandem with the red hotspot to encode the three axes of end-effector orientation. The color of the hotspot on the top-left encodes the gripper open/close state. The system is robust to distractors and changing lighting conditions. We study the problem of mapping natural language instructions and multiple posed RGBD observations to robot actions, with specific focus on quasi-static manipulation tasks that can be decomposed into a sequence of end-effector keyframes. This category encompasses a wide variety of tasks such as pick-and-place, opening/closing doors and containers, manipulating buttons, valves, switches, and more. Building robust policies that solve such tasks in unseen environments remains an open challenge that could enable numerous industrial and household applications, from cleaning and sorting robots to machine tending. VLAs have recently demonstrated successful generalization to concepts unseen in the robotics training data [ 1 , 2 , 3 , 4 ] , such as manipulating novel objects based on language instructions. While achieving generalization breakthroughs, they require massive training datasets [ 3 , 5 , 4 ] and typically accept a single RGB view input. As a result, despite the large amount of training data, the resulting systems remain sensitive to variations in camera and robot poses, which hurt their adaptability to new applications [ 6 ] . They also lack explicit visual reasoning when predicting actions (often as LLM tokens), constraining their ability to perform precise, generalizable 3D spatial reasoning. In contrast, 3D-aware keyframe based policies learn effectively from few demonstrations and generalize well to novel camera poses and object placements [ 7 , 8 , 9 , 10 ] as confirmed in the camera perturbation evaluation study by Pumacay et al. 2024 . This success stems from a 3D scene representation within the model, such as a voxel map [ 7 ] , canonical orthographic views [ 8 ] , or point clouds [ 10 ] . However, unlike VLAs, these systems overfit to the training scenes and objects, failing to accept instructions that refer to new, previously unseen objects. We propose OG-VLA (Orthographic-image Generation Vision-Language-Action model), a novel robot policy architecture that combines the generalization strengths of VLAs with robustness of 3D-aware policies. OG-VLA uses LLMs and image generation to map posed RGBD images and language to 6-DOF end-effector pose keyframes in a sequence. The system comprises four components: a point cloud renderer that renders a scene reconstruction to canonical orthographic views, a vision backbone that encodes these views into visual embeddings, an LLM that predicts action tokens, and an image diffusion model that decodes these action tokens to predict actions on each of the orthographic views through image generation, which we decode to final 3D poses. The LLM and image diffusion models are trained end-to-end, so that they work together to produce consistent and precise predictions required for robotic manipulation. Figure 1 illustrates our method. We conduct simulation and real-robot experiments. In simulation, the Arnold [ 12 ] benchmark tests generalization to unseen objects and environments, while Colosseum [ 11 ] evaluates robustness to variations in camera poses, object poses, colors, and distractors. We show significantly improved performance on generalization tests of both benchmarks, achieving state-of-the-art on Arnold [ 12 ] . In real-world experiments, we show our method’s ability to learn manipulation tasks from as few as 3 to 5 demonstrations, highlighting its suitability for kinesthetic teaching and rapid adaptation to new domains. We also present a detailed study of our model architecture design choices, providing insights into its functioning as well as potential future improvements. 2 Related Work Large Language Models have seen an explosion in research on their use-cases in robotics, such as task planning [ 13 , 14 , 15 ] , reward generation for reinforcement learning [ 16 , 17 ] , and interfacing with vision models [ 18 ] . However, although these methods generalize at a high level, they assumed access to low-level skills such pick, place, open, and close, along with perception and scene representation systems. These skills are hard to build, and their development is in line with the goals of this work. 3D-aware keyframe-based multi-task policies have shown the ability to learn complex manipulation behaviors from as little as ten demonstrations per task. Examples of these works include PerAct [ 7 ] based on voxel grids, RVT [ 8 , 9 ] based on orthonormal views, and Act3D [ 10 ] based visual feature point clouds as the scene representation. These systems significantly improve upon image-based policies [ 19 , 20 ] in the amount of data they require and their robustness to new object placements at test-time. However, these systems have been trained from scratch on specific tasks, robots, and environments, and struggle to generalize to new objects and scenes, or instructions. Vision Language Action models (VLAs) leverage large-scale prior knowledge from LLMs and vision foundation models to create robot policies that generalize to new concepts and objects at test time [ 1 , 21 , 2 , 3 , 22 , 4 ] . However, these systems require huge amount of demonstration data to train. For example, Kim et al. 2024 use over 900k demos from the Open-X Embodiment dataset [ 23 ] . Despite the large quantity of data, the resulting systems are sensitive to changes in the 3D environment, such as different