One-Shot Cross-Geometry Skill Transfer through Part Decomposition

Skye Thompson, Ondrej Biza, George Konidaris · 2026

Read One-Shot Cross-Geometry Skill Transfer through Part Decomposition by Skye Thompson, Ondrej Biza, George Konidaris (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15455.txt.

Introduction

Thompson, Biza, and Konidaris teach robots to transfer a skill to new object shapes from a single demonstration by splitting objects into semantic parts. Data-efficient generative shape models carry interaction points from the demo object's parts to matching parts of a novel object, and an auto-built objective aligns those points on skill-relevant parts. The method succeeds one-shot across varied skills and geometries in simulation and on real robots.

Catalog record

CREATORSkye Thompson, Ondrej Biza, George Konidaris
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37336
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15455

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

One-Shot Cross-Geometry Skill Transfer through Part Decomposition Skye Thompson, Ondrej Biza, George Konidaris Abstract Given a demonstration, a robot should be able to generalize a skill to any object it encounters—but existing approaches to skill transfer often fail to adapt to objects with unfamiliar shapes. Motivated by examples of improved transfer from compositional modeling, we propose a method for improving transfer by decomposing objects into their constituent semantic parts. We leverage data-efficient generative shape models to accurately transfer interaction points from the parts of a demonstration object to a novel object. We autonomously construct an objective to optimize the alignment of those points on skill-relevant object parts. Our method generalizes to a wider range of object geometries than existing work, and achieves successful one-shot transfer for a range of skills and objects from a single demonstration, in both simulated and real environments I INTRODUCTION Given a demonstration of a manipulation skill on a given object, a robot should be able to generalize that skill to novel objects—even objects that that vary substantially in shape from those it has seen before. But existing approaches fail to generalize to object geometries substantially outside their training context, especially when few demonstrations or object examples are available. Few-shot imitation learning methods that attempt to leverage geometric priors [ 18 ] [ 2 ] have shown some success in enabling transfer, but defining a prior that captures the full range of variations in object shape is nontrivial, resulting in misprediction and transfer failure if that prior is misaligned with an object instance. Successful skill adaptation requires the ability to both identify the features of a novel object that are salient for a given skill, and to successfully adapt that skill policy to the novel object using those features. Object geometric features are complex, high dimensional, and highly variable even within a category. Those features can also have a complex relationships to successful skill execution—for example, the trajectory needed to pour from a teapot with a long spout and short handle may differ significantly from a teapot with a short spout and a long handle. Existing skill learning approaches fail to capture this complexity. How can we better represent and condition on objects’ geometric features to enable skill transfer to a wider variety of objects? Fig. 1 : Using a decomposed object representation, parts-based shape warping generalizes a manipulation skill to the widest range of object geometries from the fewest training examples. We show results on a simulated mug-on-rack task, with a subsample of tested mugs enlarged for comparison. Motivated by examples of improved transfer from compositional modeling [ 8 ] [ 4 ] , we explore a method for improving transfer performance using decomposition—representing objects as compositions of independent representations, rather than a single, monolithic representation. Existing work [ 11 ] [ 15 ] presents us with a natural decomposition for many objects and skills: part decomposition, representing part shape and relationships independently. Rather than model each object monolithically, we instead model each object as reconstructed from models of its segmented constituent parts—for example, modeling a handle, body, lid, and spout, rather than a monolithic teapot. This decomposition improves transfer in two ways; first, it disentangles the learned feature representation of different parts from the relationship between those parts, which enables more accurate shape reconstruction and feature identification. And second, it enables skill generalization to a wider range of objects by modeling the relationships between object parts independently, adapting effectively when those part relationships change. To evaluate this approach, we propose a method for parts-based feature transfer and skill conditioning that adapts a demonstrated manipulation skill to a range of novel objects of different shapes. We leverage existing semantic segmentation models [ 7 ] to decompose objects into a fixed set of part categories. Given a single demonstration of a skill, our approach identifies relevant features for each part of each object in the demonstration. We transfer the part features to the novel object using generative shape models, extending existing work for transforming object features via shape warping [ 2 ] . These transformed features are then used to autonomously compose an optimization objective in the novel scene that adapts the demonstrated skill with no additional training. We compare performance on two benchmark tasks in simulation—placing a mug on a rack, and a bowl on a mug—with previous work that does not leverage part decomposition, and existing work utilizing neural descriptor fields with and without locally conditioned features [ 18 ] [ 4 ] (a different approach leveraging a similar parts-focused insight). We compare over a range of object geometries and provided demonstrations. Part decomposition allows us to successfully transfer a skill to a wider variety of objects than existing methods. We also evaluate this approach in real world pouring and placing tasks, using real objects from categories not well represented in existing benchmark datasets like ShapeNet [ 3 ] , demonstrating that this policy representation can achieve accurate transfer from limited examples. II Skill Transfer from Demonstration We consider the task of manipulating a rigid object A into a specific pose in relation to a rigid object B . We define a successful skill execution as achieving an SE(3) transformation T_{AB} , in world frame {W} , that transforms A from its initial position to a final pose relative to B that achieves the task goal, such that task cost C(A,B,T_{AB})=0 . Given a demonstration of a skill performed on these two objects, we want to transfer that demonstration to two previously unseen test objects from the same categories, A^{\prime} and B^{\prime} . These may differ in both initial pose and geometry from the demonstration objects, impacting the transformation T_{A^{\prime}B^{\prime}} required to achieve task success. We assume this transformation is a function of relevant shape features grounded in the object’s SE(3) reference frame.Concretely, we say there exists a mapping A\rightarrow I^{(A)} , where I^{(A)} refers to a set of features grounded to the object’s geometry. Each feature is grounded by associating a features i^{(A)}\in I^{(A)} with a point (x^{(A)}_{i},y^{(A)}_{i},z^{(A)}_{i}) in the object’s reference frame. This definition covers both point features, like key points [ 10 ] or contact points [ 16 ] , and semantic or shape features associated with a point or specific SE(3) transformation [ 18 ] , [ 2 ] . Given a demonstration, we can map these features to a transformation that successfully completes the skill, \pi(I^{(A)},I^{(B)})=T_{AB} , which can then be achieved by a motion planner or other fixed controller. Our goal is to capture a mapping \pi that transfers to novel objects, meaning it also captures \pi(I^{(A^{\prime})},I^{(B^{\prime})})=T_{A^{\prime}B^{\prime}} such that C(A^{\prime},B^{\prime},T_{A^{\prime}B^{\prime}})=0 . II-A Decomposition and Warping for Skill Transfer Warping is a class of solutions to the problem of skill transfer that assume there is a correspondence between the transformation of features observed in different environments f(I^{(A)},I^{(B)})=(I^{(A^{\prime})},I^{(B^{\prime})}) and the transformation between policies that succeed in those environments g(T_{AB})=T_{A^{\prime}B^{\prime}} . (We will shorten f(I^{(A)},I^{(B)}) as f(A,B) for brevity, assuming that I^{(A)} is fully determined by object geometry A ). Given f(A,B) and a functional mapping of the feature transformation to the policy transformation, f\rightarrow g , we can then predict T_{A^{\prime}B^{\prime}} given (A,B) , (A^{\prime},B^{\prime}) , and T_{AB} —which constitute

Read the complete text