LACE: Latent Visual Representation for Cross-Embodiment Learning

Yoo Sung Jang, Kanchana Ranasinghe, Cristina Mata, Yichi Zhang, Jorge Mendez-Mendez, Michael S. Ryoo · 2026

Read LACE: Latent Visual Representation for Cross-Embodiment Learning by Yoo Sung Jang, Kanchana Ranasinghe, Cristina Mata, Yichi Zhang, Jorge Mendez-Mendez, Michael S. Ryoo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16743.txt.

Introduction

Jang and colleagues propose LACE, a framework that bridges the visual gap between human and robot hands for learning from human demonstrations. Self-supervised backbones encode rich object semantics but fail to match human and robot hands. LACE aligns human and robot representations in latent space using shared body-part correspondences as sparse supervision, from forward kinematics. One robot demo suffices for training. In zero-shot transfer, LACE-DINO beats DINO by a large margin, about 65%, with gains when data is scarce and in new settings.

Catalog record

CREATORYoo Sung Jang, Kanchana Ranasinghe, Cristina Mata, Yichi Zhang, Jorge Mendez-Mendez, Michael S. Ryoo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50509
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16743

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LACE: Latent Visual Representation for Cross-Embodiment Learning Yoo Sung Jang, Kanchana Ranasinghe, Cristina Mata, Yichi Zhang, Jorge Mendez-Mendez, Michael S. Ryoo Abstract Cross-embodiment learning from human demonstrations is hindered by the visual gap between human and robot embodiments. While self-supervised learning (SSL) backbones encode rich inter-class semantics of general objects, we show they fail to establish correspondence between human and robot hands. We propose LACE, a framework that aligns human and robot visual representations in the latent space of these backbones by leveraging correspondences between shared body parts across embodiments as sparse supervision. These annotations can be automatically obtained via forward kinematics, and single robot demonstration is sufficient to train the model. Our semantic alignment loss matches distributions incurred by corresponding features, lifting patch-level supervision to semantic-level alignment, while a Gram loss preserves pretrained feature quality. This alignment enables robot policies to leverage abundant human data when robot demonstrations are scarce: in zero-shot transfer, policies using LACE-DINO outperform those using DINO by a large margin (65%), with consistent gains in low-data regimes and out-of-distribution environments. Our code will be released publicly. Figure 1: We present LACE, a framework for cross-embodiment visual representation alignment. Across embodiments, they share semantically similar body parts. Leveraging this, LACE minimizes the visual gap across embodiments in latent feature space. Keywords Cross-Embodiment Learning \cdot Dexterous Manipulation \cdot Robot Learning 1 Introduction Data scarcity is a long-standing challenge in robot learning. Despite recent efforts to collect large-scale real world datasets [ 42 , 30 , 59 ] , such data remains limited to non-dexterous robot arms. While there is growing interest in scaling datasets for dexterous hands, they have yet to reach comparable size or diversity. Instead, many works aim to transfer behaviors from human hand demonstrations [ 48 , 50 ] , as human video data is readily available on the internet. However, human and robot hands are visually distinct, creating a bottleneck for smooth domain transfer. A growing body of work addresses this visual gap across embodiments [ 33 , 11 , 35 , 34 , 51 ] , but these methods often focus on modifying representations in pixel space. These pixel-space methods have several drawbacks: they discard embodiment-specific visual information [ 33 , 21 ] , require additional image processing at inference time [ 20 , 51 , 60 ] , or are limited to a specific target embodiment [ 11 , 36 , 34 ] . We propose to bridge the cross-embodiment visual gap in latent space. Self-supervised learning (SSL) backbones have become the de facto visual representation in modern policies [ 32 , 6 , 7 , 71 , 48 , 56 ] , which makes latent-space alignment a natural fit. Our approach bypasses the aforementioned limitations of pixel-space methods. Moreover, it allows us to leverage the semantic structure that pretrained backbones already encode towards cross-embodiment alignment during training. As a concrete instantiation of our method, we fine-tune DINOv3 (hereafter DINO) using an existing keypoint-annotated human dataset and a small set of robot images whose annotations are obtained automatically from forward kinematics. We supervise alignment through correspondence between shared body parts across embodiments (e.g., human thumb to robot thumb). Our method leverages the semantic priors that pretrained backbones already encode, both to lift this keypoint-level supervision to semantic-level alignment and to regularize training. We term this framework L atent A lignment for C ross- E mbodiment (LACE) and denote backbones fine-tuned under this paradigm by prefixing LACE (e.g., LACE-DINO); while we focus on DINO for its strength and popularity in robot learning, the approach readily transfers to other self-supervised backbones that encode semantic structure. Our main contributions are: • We identify that the visual gap between human and robot embodiments propagates to latent visual embeddings and propose a fine-tuning approach that leverages the backbone’s inherent semantic priors to minimize this gap. • We achieve cross-embodiment semantic alignment using frames sampled from a single robot demonstration - or even a single keypoint-annotated frame - while preserving the general semantic features, avoiding catastrophic forgetting. • We empirically validate in real-world experiments that latent-space alignment benefits policy learning in low robot data regimes and enables generalization to unseen environments. 2 Related Works Visual Gap in Cross-Embodiments Learning Robot learning across embodiments is challenged by the visual appearance gap between the source and target embodiment. One approach is to sidestep the visual gap using wrist-mounted cameras, which exclude the embodiment from visual observation [ 12 , 52 , 8 , 66 , 27 , 24 ] . However, wrist views omit crucial context such as visual robot state and holistic scene layout. Several recent methods handle the visual gap by modifying images directly in pixel space [ 28 , 33 , 11 , 36 , 34 , 15 ] . These approaches use techniques such as segmentation masks or inpainting to make observations from different embodiments appear visually similar. However, pixel-space modifications have key drawbacks: they discard embodiment-specific information such as morphology, require additional image processing during deployment, and often the policy is committed to a specific target embodiment. Other work provides embodiment-agnostic visual cues to guide the policy [ 20 , 60 , 50 , 41 , 26 , 19 ] , but still requires additional compute during inference. An alternative direction operates in latent space: recent work observes that shared representations can emerge through co-training on mixed human-robot data [ 29 , 47 , 37 , 31 , 65 ] . However, these methods require large and diverse datasets, or temporally aligned demonstrations across embodiments which need to be carefully collected. Our method does not require a temporally or spatially aligned dataset nor a large dataset, yet we acheive the latent representation alignment leveraging the encoded semantics of a pretrained SSL backbone. We fine-tune the backbone using samples from as few as one demonstration, achieving cross-embodiment alignment without large datasets or inference overhead. Self-Supervised Visual Backbones Self-supervised models such as DINO-v2 & v3, SigLIP and V-JEPA2 [ 43 , 55 , 58 , 2 ] are widely adopted as visual backbones in modern vision-language-action models [ 32 , 53 , 14 , 48 , 64 ] . Trained on internet-scale data, these models produce dense, locality-aware, and semantically rich features, which have proven useful for downstream tasks including policy learning in robotics. However, these properties do not fully transfer to cross-embodiment learning settings in robotics, especially with dexterous manipulators [ 67 , 63 , 62 , 68 , 23 , 17 , 16 , 69 , 18 , 5 ] . Self-supervised representations are typically pretrained on internet images that contain less robotic embodiments or manipulation interactions. As we show in Section 3 , pretrained DINO (self-supervised) features fail to establish reliable semantic correspondence between human and robot hands despite their functional and structural similarities. While robotics-specific self-supervised encoders exist [ 40 , 49 , 38 , 25 ] , their ability to quickly adapt to embodiments unseen during pretraining remains underexplored. Our work addresses this gap by explicitly aligning features across embodiments using only few samples, while preserving the rich semantic properties of the pretrained SSL backbone. Table 1: Keypoint alignment metrics comparison. We evaluate keypoint alignment on different embodiment pair settings. DINO

Read the complete text