No More Blind Spots: Learning Vision-Based Omnidirectional Bipedal Locomotion for Challenging Terrain
Mohitvishnu S. Gadde, Pranay Dugar, Ashish Malik, Alan Fern · 2025
Read No More Blind Spots: Learning Vision-Based Omnidirectional Bipedal Locomotion for Challenging Terrain by Mohitvishnu S. Gadde, Pranay Dugar, Ashish Malik, Alan Fern (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11929.txt.
Introduction
Mohitvishnu Gadde and co-authors give a two-legged robot eyes in every direction and teach it to walk rough ground without rendering a fortune in simulation. Agile bipeds need omnidirectional terrain sensing, but simulating surround depth images is so costly that learning in simulation stalls. Their trick pairs teachers and students.
A robust blind controller and a privileged teacher policy supervise a vision-based student, trained on noise-augmented terrain data instead of freshly rendered images. A new augmentation scheme speeds training up to tenfold. The student walks in all directions over challenging terrain. Robots that can sidestep and back out of clutter, not just march forward, belong in real buildings.
Catalog record
| CREATOR | Mohitvishnu S. Gadde, Pranay Dugar, Ashish Malik, Alan Fern |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-11929
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
No More Blind Spots: Learning Vision-Based Omnidirectional Bipedal Locomotion for Challenging Terrain Mohitvishnu S. Gadde, Pranay Dugar, Ashish Malik, Alan Fern Abstract Effective bipedal locomotion in dynamic environments, such as cluttered indoor spaces or uneven terrain, requires agile and adaptive movement in all directions. This necessitates omnidirectional terrain sensing and a controller capable of processing such input. We present a learning framework for vision-based omnidirectional bipedal locomotion, enabling seamless movement using depth images. A key challenge is the high computational cost of rendering omnidirectional depth images in simulation, making traditional sim-to-real reinforcement learning (RL) impractical. Our method combines a robust blind controller with a teacher policy that supervises a vision-based student policy, trained on noise-augmented terrain data to avoid rendering costs during RL and ensure robustness. We also introduce a data augmentation technique for supervised student training, accelerating training by up to 10 times compared to conventional methods. Our framework is validated through simulation and real-world tests, demonstrating effective omnidirectional locomotion with minimal reliance on expensive rendering. This is, to the best of our knowledge, the first demonstration of vision-based omnidirectional bipedal locomotion, showcasing its adaptability to diverse terrains. I INTRODUCTION Legged locomotion has recently been a central objective of the robotics research community due to the capability of such robots to traverse across natural and unstructured terrains [ 1 , 2 , 3 , 4 ] . Over the past decade, learning-based approaches, particularly blind locomotion controllers, have demonstrated remarkable progress in enabling stable walking and running across a variety of environments [ 5 , 6 ] . These blind policies capitalize on proprioceptive feedback to maintain balance and momentum, forming a strong foundation for real-world deployment [ 7 ] . However, despite their robustness on well-modeled or homogeneous surfaces, blind controllers often struggle in the face of unforeseen obstacles or complex, uneven terrains. To address these limitations, recent research has focused on vision-based locomotion controllers [ 8 , 9 ] , where exteroceptive sensors such as cameras or depth sensors provide crucial information about the environment. By analyzing visual inputs, these controllers adapt to the changes in terrain and proactively adjust foot placements [ 10 ] , thereby improving stability and efficiency. Although such vision-based methods enable new possibilities for agile and adaptive movement, they also introduce challenges in perception, sim-to-real transfer, and high-dimensional policy learning [ 11 ] . Fig. 1 : The omni-direction robot locomotion controller is trained in simulation via student-teacher learning to proactively adjust the gait to challenging terrain conditions based on the input from 4 depth cameras. This controller is then transferred to the real robot. Training end-to-end vision-based locomotion controllers is notoriously challenging, particularly when using reinforcement learning (RL), due to the high computational cost of simulating realistic visual inputs. Rendering depth or RGB images for complex environments is expensive [ 12 ] , which slows training and limits throughput. These difficulties are amplified in the omnidirectional setting, where the volume of visual data increases by at least a factor of four, placing even greater demands on simulation and processing. In addition to rendering cost, sim-to-real transfer introduces further complexity: policies must be robust to real-world noise, actuator delays, and the absence of privileged information often used during training. While most prior work on vision-based bipedal locomotion has focused on forward [ 13 ] or constrained directional movement to simplify perception and control, practical applications demand omnidirectional agility, posing a significant challenge for scalable training and deployment of vision-based policies [ 14 ] . To address these challenges, we propose a novel framework for vision-based omnidirectional bipedal locomotion that balances computational efficiency with robust real-world performance. The framework is explicitly designed to minimize reliance on computationally expensive depth rendering during training. It achieves this through three key strategies. First, we initialize training with a stable, pre-trained blind locomotion policy that provides basic balance and movement capabilities, serving as a reliable backbone for the vision-based controller. Second, we employ a student-teacher distillation approach enhanced with DAGGER [ 15 ] , where a privileged teacher policy is trained via simulation-intensive reinforcement learning using low-cost height maps and environment state information. The vision-based student then learns from this teacher through supervised imitation, using rendered depth images only for the student policy. Third, we introduce a data augmentation technique during student training that significantly increases the effective training set without additional rendering. Together, these components enable efficient training of vision-aware omnidirectional policies within practical time and resource budgets. Our experimental results demonstrate that our proposed framework leads to significantly faster training than conventional approaches while maintaining competitive performance in both simulation and real-world scenarios. In summary, the key contributions of this work are as follows: • We present, to the best of our knowledge, the first framework for robust, vision-guided omnidirectional bipedal locomotion. • We propose an instantiation of student-teacher training specifically targeted at reducing the impractical overhead of directly training with depth rendering in the reinforcement-learning loop. • We propose an innovative data augmentation strategy during student training, where duplicating data buffers and varying velocity commands enhances dataset diversity at a minimal additional computational cost. This approach reduces rendering overhead and shortens overall training time, while improving the policy’s generalization to real-world scenarios. II Related Work II-A Reinforcement Learning (RL) for Bipedal Robots In recent years, RL-based controllers have been extensively researched for legged locomotion [ 7 , 16 , 5 ] . These controllers are usually trained in simulation using model-free RL approaches, which leverage the robot’s proprioception, enabling legged robots to traverse various terrains without taking any explicit gait input to achieve locomotion. However, despite the advances, these blind locomotion policies primarily rely upon the proprioceptive feedback from the robot to learn a policy that is able to achieve stability and maintain balance during locomotion in the presence of moderate disturbances. This allows for locomotion on well-structured surfaces with limited amounts of ground and external disturbances. Unfortunately, blind RL-based locomotion controllers struggle with unknown obstacles and significant terrain variations as they cannot sense or anticipate the changes in the environment they are operating under before making foot placements. This limitation often results in suboptimal locomotion performance and requires excessive corrective inputs to traverse such environments to avoid failures. Vision-based locomotion has been explored as a means to provide exteroceptive sensing to address these challenges, allowing these legged robots to perceive and interpret the terrain features and traverse varying terrain. . II-B Vision-Based Legged Locomotion Vision-based locomotion has been extensively researched for quadrupedal robots [ 17 , 1 , 3 , 4 ] and, to a lesser extent, for bipedal robots [ 13 , 14 ] . These methods typically use RL to train and demonstrate sim-to-real transfer of these vision-based locomotion controllers. Elevation maps or height scans in the global frame of reference have been used as the observed perceptive input to train the RL policy. For example, bipedal robots [ 13 ] and quadrupedal robots [ 18 , 19 , 20 , 21 ] utilize these height map scans around each foot. While several other approaches [ 18 , 19 ] employ a uniformly structured elevation map that is centered at the robot frame. By assuming the availability of height maps at