LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations

Weikang Wan, Jiawei Fu, Xiaodi Yuan, Yifeng Zhu, [0.5ex] University of California San Diego · 2025

Read LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations by Weikang Wan, Jiawei Fu, Xiaodi Yuan, Yifeng Zhu, [0.5ex] University of California San Diego (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17547.txt.

Introduction

Weikang Wan, Jiawei Fu, Xiaodi Yuan, Yifeng Zhu, and Hao Su presented LodeStar in 2025, a system that teaches robots long, skilled tasks from a few human examples. Such tasks need both fine hand control and smooth linking of many steps. Gathering enough real examples to train a robot is slow and costly.

LodeStar first splits each human example into clear skills, using ready-made foundation models. It then creates many varied fake examples in simulation through trial-and-error learning. A single policy learns to chain the skills in order and adapt to changes in the scene. On three hard real-world tasks, it beat earlier methods in success and robustness, the authors report.

Catalog record

CREATORWeikang Wan, Jiawei Fu, Xiaodi Yuan, Yifeng Zhu, [0.5ex] University of California San Diego
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17547

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations Weikang Wan, Jiawei Fu, Xiaodi Yuan, Yifeng Zhu, [0.5ex] University of California San Diego Abstract Developing robotic systems capable of robustly executing long-horizon manipulation tasks with human-level dexterity is challenging, as such tasks require both physical dexterity and seamless sequencing of manipulation skills while robustly handling environment variations. While imitation learning offers a promising approach, acquiring comprehensive datasets is resource-intensive. In this work, we propose a learning framework and system LodeStar that automatically decomposes task demonstrations into semantically meaningful skills using off-the-shelf foundation models, and generates diverse synthetic demonstration datasets from a few human demos through reinforcement learning. These sim-augmented datasets enable robust skill training, with a Skill Routing Transformer policy effectively chaining the learned skills together to execute complex long-horizon manipulation tasks. Experimental evaluations on three challenging real-world long-horizon dexterous manipulation tasks demonstrate that our approach significantly improves task performance and robustness compared to previous baselines. Videos are available at lodestar-robot.github.io . Keywords: Dexterous Manipulation, Imitation Learning, Sim-to-Real Figure 1: We present LodeStar , a framework for learning robust long-horizon dexterous manipulation from a few human demonstrations. LodeStar segments demonstrations into sequential skills using off-the-shelf foundation models, and augments each skill via simulation-based residual reinforcement learning. The synthesized skill data is co-trained with real-world data, and a Skill Routing Transformer composes the learned skills into a robust autonomous policy capable of completing complex real-world tasks. 1 Introduction Developing multi-fingered robotic systems capable of long-horizon manipulation with human-level dexterity has been a longstanding goal in robotics research. Consider everyday activities such as watering a plant with a spray bottle as shown in Fig. 1 : this requires not only physical dexterity to insert and twist the nozzle, lift and hold the bottle upright, and press the trigger, but also the ability to seamlessly sequence these distinct behaviors while robustly handling environment variations. Imitation learning (IL) from human demonstrations [ 1 , 2 , 3 ] offers an effective pathway for teaching robots such behaviors. However, for such tasks, acquiring massive, comprehensive, and diverse datasets necessary to ensure policy robustness across potential perturbations and environmental variations encountered in the real world is often costly and resource-intensive [ 4 , 5 ] . This challenge motivates our investigation into methods that leverage structured data generation from limited demonstrations, aiming to enable robots to perform long-horizon dexterous manipulation tasks robustly. Prior works have explored generating large datasets from a few demonstrations using replay-based transformation with motion planning [ 6 , 7 , 8 , 9 ] . While effective, these methods are often limited to parallel grippers or fixed-hand motions. Moreover, naive composing the adapted segments can lead to low-quality or low-diversity demonstrations, hindering generalizable policy learning. Another direction leverages reinforcement learning (RL) in simulation to synthesize diverse data. Recent advances have enabled dexterous skills such as grasping [ 10 , 11 , 12 ] and in-hand reorientation [ 13 , 14 , 15 ] , as RL exploration can discover varied and robust behaviors. However, these results are typically limited to single-stage tasks, rely on hand-tuned simulators and rewards, and remain sensitive to the sim-to-real gap. In parallel, several works decompose long-horizon manipulation into sequential skills and chain them to solve complex tasks [ 16 , 17 , 18 , 19 , 20 , 21 ] , reducing both the complexity of data generation and policy learning. Yet, smooth and reliable skill transfer remains challenging due to state distribution mismatches at skill boundaries [ 22 ] , often requiring additional regularization [ 23 , 24 ] or optimization [ 25 ] techniques, which may destabilize policy learning. Despite these advances, robust long-horizon dexterous manipulation from limited data still faces two key challenges: (1) how to automatically segment tasks into semantically meaningful skills with appropriate initial and terminal state distributions; and (2) how to generate diverse training data to robustify individual skills, and effectively chain them into a coherent policy that completes the full task. To address these challenges, we propose a structured and scalable framework LodeStar that decomposes long-horizon dexterous manipulation into three stages: skill segmentation, synthetic data generation for robust skill learning, and skill composition via a Skill Routing Transformer (SRT) policy. We first segment demonstrations into sequential manipulation skills and intermediate transition motions by leveraging foundation models (VFMs, VLMs) [ 26 , 27 , 28 ] to extract visual, spatial, and contact cues directly from raw videos, avoiding manual annotation or predefined primitive skills. For each manipulation skill, we train a robust policy via residual reinforcement learning in simulation environments generated through real-to-sim techniques with augmentation and domain randomization. The learning process is grounded in a small number of real demonstrations, ensuring task relevance and avoiding reward engineering. We further synthesize diverse and physically feasible transition trajectories between skills, eliminating the need for explicit goal specification or slow execution speed of real-world motion planning. Finally, we train a SRT policy on the generated data to compose the learned skills, enabling coherent execution of the full long-horizon task. Our key contribution is a scalable learning framework LodeStar for robust long-horizon dexterous manipulation from a few human demos. By combining automatic segmentation via foundation models, real-to-sim synthetic data generation for skill acquisition, and skill composition for sim-to-real deployment, LodeStar enables complex manipulation sequences without requiring extensive task-specific data or manual engineering effort. We validate its effectiveness on three challenging real-world tasks, showing that it outperforms SOTA baselines by an average of 25% success rate. 2 Related Work 2.1 Dexterous Manipulation Dexterous manipulation remains a long-standing and significant challenge in robotics [ 29 , 30 , 31 , 32 ] , primarily due to the high degrees of freedom involved. Traditional planning and control approaches [ 33 , 34 , 35 , 36 ] often rely on simplified system models, which struggle to scale to complex, long-horizon tasks. Recent advances have demonstrated the effectiveness of learning-based methods for specific dexterous skills such as grasping [ 11 , 10 , 37 , 12 ] , in-hand reorientation [ 38 , 14 , 15 ] , and tool use [ 39 , 40 , 41 , 42 , 43 ] . However, these methods typically target isolated skills rather than integrated, sequential behaviors. In contrast, we tackle the more challenging task of sequencing multiple dexterous skills [ 25 ] , which requires both fine-grained control and temporal coordination. 2.2 Synthetic Data Generation for Robotic Manipulation Synthetic simulation data have been used to alleviate the burden of real-world data collection and improve policy generalization in robotic manipulation. Recent studies [ 44 , 45 , 46 , 47 , 48 , 49 , 50 , 9 , 51 , 52 ] show that co-training policy with simulation data can greatly improve policy performance and robustness by diversifying the training distribution beyond what real-world data alone can offer. Real-to-sim approaches have been used to generate realistic and diverse simulation assets and scenes from real-world inputs via 3D reconstruction [ 53 , 54 , 55 , 56 , 57 , 58 , 59 ] , inverse graphics [ 60 ] , and foundation model-assisted generation [ 61 , 45 , 62 , 63 ] . Techniques for sim-to-real transfer

Read the complete text