Bridging Perception and Action: Spatially-Grounded Mid-Level Representations for Robot Generalization

Jonathan Yang 1 2, Chuyuan Kelly Fu 2, Dhruv Shah 2, Dorsa Sadigh 1 2 · 2025

Read Bridging Perception and Action: Spatially-Grounded Mid-Level Representations for Robot Generalization by Jonathan Yang 1 2, Chuyuan Kelly Fu 2, Dhruv Shah 2, Dorsa Sadigh 1 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06196.txt.

Introduction

Robots generalize better when they see the world in grasp sized pieces, show Jonathan Yang and co-authors in a 2025 arXiv paper. Instead of feeding a policy raw images or words, the team feeds it mid level maps that mark objects, poses, and depth, each produced by its own trained encoder.

A mixture of experts policy then blends the specialists for two handed manipulation in the real world. Success ran eleven points above a language grounded baseline and twenty four above a plain diffusion policy. Using the same maps as training targets added ten more points. The lesson: action shaped perception beats description alone.

Catalog record

CREATORJonathan Yang 1 2, Chuyuan Kelly Fu 2, Dhruv Shah 2, Dorsa Sadigh 1 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06196

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging Perception and Action: Spatially-Grounded Mid-Level Representations for Robot Generalization Jonathan Yang 1 2, Chuyuan Kelly Fu 2, Dhruv Shah 2, Dorsa Sadigh 1 2 Abstract In this work, we investigate how spatially-grounded auxiliary representations can provide both broad, high-level grounding, as well as direct, actionable information to help policy learning performance and generalization for dexterous tasks. We study these mid-level representations across three critical dimensions: object-centricity, pose-awareness, and depth-awareness. We use these interpretable mid-level representations to train specialist encoders via supervised learning, then use these representations as inputs to a diffusion policy to solve dexterous bimanual manipulation tasks in the real-world. We propose a novel mixture-of-experts policy architecture that can combine multiple specialized expert models, each trained on a distinct mid-level representation, to improve the generalization of the policy. This method achieves an average of 11 % 11\% higher success rate on average over a language-grounded baseline and a 24 % 24\% higher success rate over a standard diffusion policy baseline for our evaluation tasks. Furthermore, we find that leveraging mid-level representations as supervision signals for policy actions within a weighted imitation learning algorithm improves the precision with which the policy follows these representations, leading to an additional performance increase of 10 % 10\% . Our findings highlight the importance of grounding robot policies with not only broad, perceptual tasks, but also more granular, actionable representations. For further information and videos, please visit https://mid-level-moe.github.io . I Introduction Large pre-trained robotics models have made significant progress in recent years towards improving robotic generalization capabilities by leveraging large-scale pre-training datasets. However, these models still face challenges in adapting to slight scene variations such as different spatial locations, unseen objects, and different lighting conditions. An increasingly popular approach to address this challenge is explicitly establishing deeper connections between robot policies and the abstract patterns and relationships that govern the physical world. For example, vision-language-action models (VLAs) make an attempt to benefit from semantic and visual knowledge from vision-language models (VLMs) by fine-tuning these models with robot data. Other works use explicit mid-level representations such as low-level language instructions [ 2 ] , key-points [ 36 , 25 ] , or trajectories [ 31 , 14 , 38 ] to provide additional grounding to the robot policy. Despite the success of these methods, the generalization properties that these additional forms of grounding enable are still high-level in nature, and their benefits for simpler tasks may not necessarily transfer to more complex tasks – ones that require further dexterity or object interactions. We hypothesize that the choice of mid-level representations is highly dependent on task-specific requirements. For instance, for a robot tasked with folding a shirt, a bounding box may help locate a shirt’s general position but fails to provide actionable information on how to manipulate it. Similarly, depth-informed representations are crucial for contact-rich tasks such as handing over objects, but may be less important for overhead pick-and-place tasks. In order to give models stronger generalization capabilities for a wide variety of dexterous tasks, it is essential to explore representations that balance high-level abstraction and low-level actionable detail. These representations must not only encode spatial and geometric reasoning but also offer adaptability across diverse and dynamic environments. In this paper, we find a set of mid-level representations that enhances the adaptability of a robot policy across a wide variety of environments. We first systematically study these representations across four critical dimensions: object-centricity , or understanding of the locations and geometry of objects on the scene; motion-centricity or understanding of the future motion of the robot; pose-awareness , or understanding of spatial orientations; and depth-awareness , or understanding about three-dimensional structure and geometry. Through these experiments, we identify how different forms of spatial grounding align with task-specific requirements. We then propose a method to leverage these representations through a diffusion policy-based model conditioned on multiple experts outputting interpretable representations. We show that while different mid-level representations excel at different tasks, our method can leverage these task-specific benefits to achieve consistently higher performance on a wide range of environments. In addition, we further investigate how robot policies utilize the aforementioned representations. We find that reliance on structured signals presents a trade-off: policies that depend heavily on these representations can become more susceptible to overfitting and reduced robustness in noisy environments. To mitigate this, we introduce key architectural design choices that balance sensitivity to mid-level representations with resilience against spurious noise in these representations. Finally, we incorporate these representations as additional training signal. We refer to the alignment of the demonstrations with these mid-level representation as self-consistency , which can provide a weighting scheme for weighted imitation learning – upweighting data with self-consistent representations. This approach further refines the policy’s ability to execute mid-level plans with greater precision and reliability. Our policy, Mid-Level MoE, achieves a 24\% higher success rate than a standard diffusion policy baseline and an 11\% improvement over a baseline using language representations across a series of bimanual, dexterous tasks. Furthermore, our weighted imitation learning method enhances the policy’s reliance on mid-level representations while preserving robustness to perturbations, resulting in a 10\% higher success rate compared to standard training. These results highlight that a deeper understanding of robot grounding can lead to significant improvements in robot success in dexterous, bimanual environments. II Related Works Training generalist robot policies has been typically approached as a multi-task learning problem, where a single machine learning model is optimized for different behaviors and objectives. Many prior works have involved scaling up the breadth and diversity of robot data [ 8 , 12 , 22 , 26 , 40 , 21 , 30 ] . A key challenge with the multi-task policy learning regime is in obtaining policies that generalize to new objects, task variants, environmental factors and so on. Towards this, a significant body of work has focused on generalizing robot policies to grasp new objects [ 32 , 28 , 19 ] . In this work, we are interested in achieving a similar form of generalization for dexterous manipulation [ 6 , 39 ] . Prior works in learning multi-task dexterous policies have struggled with generalizing to entirely new objects due to the high-dimensional observation and action spaces. While the typical recipe to obtaining generalizable policies is to scale robot data, collecting such data remains prohibitively expensive. A promising alternative is to introduce structure into the end-to-end pixels-to-actions mapping. To provide robot models with a greater understanding of commonalities in robot tasks, planning, and behavior, several previous works have considered conditioning robot policies with higher-level representations of robot behavior [ 33 , 27 , 16 , 37 ] . To increase the interpretability and controllability of robot policies, many works have instead conditioned on explicit representations. These representations typically been specified either through goal images [ 29 , 9 ] , video demonstrations [ 35 , 11 ] or language [ 13 , 20 , 41 ] . While many earlier works have used higher-level conditioning information at the task specification level, recent works have works towards getting robots to achieve more specific goals, often specified in language [ 17 , 1 , 18 , 3 ] . One potential drawback to the hierachical learning framework is

Read the complete text