TReF-6: Inferring Task-Relevant Frames from a Single Demonstration for One-Shot Skill Generalization
Yuxuan Ding Shuangge Wang Tesca Fitzgerald · 2025
Read TReF-6: Inferring Task-Relevant Frames from a Single Demonstration for One-Shot Skill Generalization by Yuxuan Ding Shuangge Wang Tesca Fitzgerald (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00310.txt.
Introduction
Yuxuan Ding, Shuangge Wang, and Tesca Fitzgerald present TReF-6 in 2025, a method that lets robots generalize a skill from a single demonstration. One-shot learning is hard without a transferable, interpretable spatial frame. Their method infers a simplified six-degree-of-freedom task-relevant frame from one trajectory, which is the path a hand or tool follows.
An influence point, drawn purely from trajectory geometry, sets the frame's origin. That frame then parameterizes a Dynamic Movement Primitive, a standard motion model, beyond plain start-goal copying. A vision-language model grounds the frame in meaning, and a grounding tool locates it in new scenes. Real manipulation tests show functionally consistent transfer.
Catalog record
| CREATOR | Yuxuan Ding Shuangge Wang Tesca Fitzgerald |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-00310
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TReF-6: Inferring Task-Relevant Frames from a Single Demonstration for One-Shot Skill Generalization Yuxuan Ding Shuangge Wang Tesca Fitzgerald Abstract Robots often struggle to generalize from a single demonstration due to the lack of a transferable and interpretable spatial representation. In this work, we introduce TReF-6 , a method that infers a simplified, abstracted 6 DoF T ask- Re levant F rame from a single trajectory. Our approach identifies an influence point purely from the trajectory geometry to define the origin for a local frame, which serves as a reference for parameterizing a Dynamic Movement Primitive (DMP). This influence point captures the task’s spatial structure, extending the standard DMP formulation beyond start-goal imitation. The inferred frame is semantically grounded via a vision-language model and localized in novel scenes by Grounded-SAM, enabling functionally consistent skill generalization. We validate TReF-6 in simulation and demonstrate robustness to trajectory noise. We further deploy an end-to-end pipeline on real-world manipulation tasks, showing that TReF-6 supports one-shot imitation learning that preserves task intent across diverse object configurations. Keywords: Spatial Reference Frames, One-Shot Imitation Learning, Dynamic Movement Primitives 1 Introduction Robots are increasingly expected to operate in dynamic, human-centered environments, whether assisting in homes [ 1 ] , collaborating in warehouses [ 2 ] , or supporting kitchen workflows [ 3 ] . These settings demand generalizable behavior: adapting motion to unseen objects, adjusting to new object placements and orientations, and aligning with different surface orientations. While humans perform such adaptations effortlessly, robots struggle to generalize skills from limited demonstrations. Since training data can never fully prevent out-of-distribution (OOD) scenarios, researchers increasingly turn to extract structural representations, such as rigid-body poses [ 4 ] or keypoints [ 5 , 6 , 7 ] , which provide a more stable foundation for generalization in manipulation tasks. Recent work has developed generalizable techniques for generalizing a task in terms of goal poses [ 8 , 9 , 10 ] , sub-goals [ 11 ] , or contact interactions [ 12 ] . There has been less work, however, on generalizing the spatial constraints encoded in the trajectory itself, such as the curvature of opening the door indicating a hinge constraint besides just the handle position. In practice, the shape of a human demonstration reflects more than just start and end points. It encodes implicit constraints, such as obstacle avoidance [ 13 ] , mechanical limitations (e.g. hinge constraints) [ 14 ] , or ergonomic preferences [ 15 ] . For any two points in space, infinitely many paths exist, yet demonstrators tend to follow specific, repeatable curves. Prior works in [ 16 ] have shown that humans opt for these repeatable trajectories due to their similarities in more condensed, latent geometric structure. We believe that inferring these latent structures could be informative in generalizing trajectories to unseen objective configurations. Furthermore, we expect that they may correspond to semantic features that are detectable by vision-language models (VLMs) [ 17 ] . This motivates our central research question: Can we use a single demonstration to infer a task-relevant spatial reference frame that is (1) semantically identifiable (anchors to scene features) and (2) functionally meaningful (preserves constraints), enabling generalization across object poses and configurations? In this work, we propose TReF-6 (Figure 1 ), a framework that infers a task-relevant, 6DoF reference frame from a single demonstration that enables simple dynamical controllers such as DMPs to generalize their motion robustly. Our contributions include: 1. Formalizing the problem of inferring a task-relevant frame as an optimization over its geometric consistency to the trajectory dynamics; 2. An efficient optimization algorithm that is robust to trajectory noise and does not rely on object priors, human labels, or dense annotations; and 3. Simulated and physical robot evaluations that demonstrate our method’s ability to generalize to new spatial variations of demonstrated tasks. Figure 1: Overview of TReF-6. Given a single demonstration, TReF-6 infers an implicit influence point, semantically grounded by a vision-language model (VLM), and extracts a 6-DoF reference frame from the segmentation provided by Grounded-SAM. With minimum assumptions, the inferred frame enables robust OOD generalization. 2 Related Work Dynamic Movement Primitives (DMPs) encode a demonstrated trajectory as a stable dynamical system [ 18 , 19 ] that converges toward a goal. DMPs can generalize the shape of a demonstrated motion to target a new goal pose, yet are functionally agnostic to the surrounding environment unless manually augmented with additional force-like terms that account for external influences [ 20 ] or goal and sub-goal related task frames [ 21 ] . Without access to an appropriate task-specific reference, DMPs are trained over just a single trajectory, so by definition they “overfit” to the environmental configuration and constraints that were present during the initial demonstration. Data-driven approaches such as TP-GMMs [ 22 ] and their extensions [ 23 ] involve conditioning a demonstrated trajectory on external task parameters. These approaches improve adaptation across different object poses, placements, and orientations, but require extensive training data (including synthetic data) to generalize effectively [ 22 , 24 ] , or assume that relevant task-relevant frames, such as those aligned with object or interaction axes, are provided or can be easily extracted [ 25 ] . Affordance learning focuses on the actionable possibilities in an environment, such as regions on an object that can be grasped, pushed, or opened [ 26 ] , capturing the physical interactions that enable task completion. Recent efforts in this field have focused on visually identifying object surfaces that support affordances based on robot-collected data [ 27 ] or human demonstration videos [ 28 , 29 ] . Others use depth data to address occlusion and collision constraints in articulated objects [ 30 ] . While these approaches have shown promising results, they involve training visual backbones with target object types or multi-view RGB inputs. In real-world scenarios where a robot must interact with unseen novel objects, especially under varying camera viewpoints with crowded objects and random occlusions, such reliance becomes a bottleneck in terms of data requirements and system robustness. One-shot imitation learning targets this challenge of adapting to novel task constraints or objects. This may involve grounding the demonstration using external context—language descriptions or videos of related tasks to make sense of the action in a broader setting [ 31 ] . Other methods focus on low-level structure, aligning object parts or motion trajectories across different scenes to find transferable patterns [ 32 , 8 , 33 ] or with rich contact information [ 12 ] . Recent work also looks for structure that is consistent across different environments, for example, regions in 3D space that consistently guide behavior across tasks [ 34 ] , or prior demonstrations retrieved based on graph-based similarity metrics [ 35 ] . These approaches enable a robot to quickly generalize its task knowledge to novel constraints and objects, and reinforce the importance of a reasoning module that correctly and efficiently identifies task-relevant features for one-shot learning [ 36 ] . Although efficient at generalization, one-shot methods still require a large amount of data at training to acquire the ability to learn-to-learn and generalize from a single new example. Compared to end-to-end one-shot policy learning approaches, DMPs offer advantages in stability, data