Scaffolding Dexterous Manipulation with Vision-Language Models

Vincent de Bakker Joey Hejna Tyler Ga Wei Lum, Onur Celik Aleksandar Taranovic Denis Blessing, Gerhard Neumann Jeannette Bohg Dorsa Sadigh · 2025

Read Scaffolding Dexterous Manipulation with Vision-Language Models by Vincent de Bakker Joey Hejna Tyler Ga Wei Lum, Onur Celik Aleksandar Taranovic Denis Blessing, Gerhard Neumann Jeannette Bohg Dorsa Sadigh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19212.txt.

Introduction

Vincent de Bakker and co-authors wrote a 2025 paper on teaching robot hands hard tasks with help from vision-language models. Skilled hand work is hard to train, since demos are scarce and rewards are fussy to design. Their trick is to let a general model sketch the job first.

Given a task line and a scene, the model picks key points like handles, then drafts rough 3D paths for hand and object. A learning policy in simulation then polishes those coarse paths into firm control. The method learned robust skills on hinged objects and meaning based tasks. It also moved to real robot hands with no human demos and no hand made rewards.

Catalog record

CREATORVincent de Bakker Joey Hejna Tyler Ga Wei Lum, Onur Celik Aleksandar Taranovic Denis Blessing, Gerhard Neumann Jeannette Bohg Dorsa Sadigh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-19212

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaffolding Dexterous Manipulation with Vision-Language Models Vincent de Bakker Joey Hejna Tyler Ga Wei Lum, Onur Celik Aleksandar Taranovic Denis Blessing, Gerhard Neumann Jeannette Bohg Dorsa Sadigh Abstract Dexterous robotic hands are essential for performing complex manipulation tasks, yet remain difficult to train due to the challenges of demonstration collection and high-dimensional control. While reinforcement learning (RL) can alleviate the data bottleneck by generating experience in simulation, it typically relies on carefully designed, task-specific reward functions, which hinder scalability and generalization. Thus, contemporary works in dexterous manipulation have often bootstrapped from reference trajectories. These trajectories specify target hand poses that guide the exploration of RL policies and object poses that enable dense, task-agnostic rewards. However, sourcing suitable trajectories---particularly for dexterous hands---remains a significant challenge. Yet, the precise details in explicit reference trajectories are often unnecessary, as RL ultimately refines the motion. Our key insight is that modern vision-language models (VLMs) already encode the commonsense spatial and semantic knowledge needed to specify tasks and guide exploration effectively. Given a task description (e.g., “open the cabinet”) and a visual scene, our method uses an off-the-shelf VLM to first identify task-relevant keypoints (e.g., handles, buttons) and then synthesize 3D trajectories for hand motion and object motion. Subsequently, we train a low-level residual RL policy in simulation to track these coarse trajectories or ‘‘scaffolds’’ with high fidelity. Across a number of simulated tasks involving articulated objects and semantic understanding, we demonstrate that our method is able to learn robust dexterous manipulation policies. Moreover, we showcase that our method transfers to real-world robotic hands without any human demonstrations or handcrafted rewards. 1 1 1 Videos and Code at https://sites.google.com/view/dexterous-vlm-scaffolding 1 Introduction Dexterous manipulation is essential for a range of real-world tasks – such as using a power-drill or twisting a door knob – which require the fine-grained control offered by human-like hands [ 2 ] . Enabling dexterous capabilities has therefore been a long-standing goal in robotics. Despite the intrinsic advantages of dexterous hands over simpler end-effectors, existing learning paradigms have struggled to cope with their complexity [ 52 ] . The prevailing approach for training generalist policies – imitation learning from demonstrations [ 5 , 45 ] – has achieved limited success with robot hands, primarily due to the challenges of collecting accurate data with dexterous hardware, resulting in a scarcity of high-quality demonstrations [ 50 , 62 ] . While alternative approaches attempt to re-target demonstrations from easier interfaces [ 28 , 21 , 67 , 69 , 53 , 74 , 49 , 15 ] , e.g., human hands, such approaches often induce irrecoverable errors for fine-grained tasks. To avoid both data scarcity and the embodiment gap, a combination of reinforcement learning (RL) and sim-to-real transfer has emerged as a promising approach by enabling large-scale experience generation [ 3 ] . However, using RL simply shifts the burden from data collection to reward design. Standard RL approaches for dexterous manipulation necessitate hand-crafting complex, task-specific reward functions. A large amount of this complexity arises from the need to guide exploration; with large action spaces, dexterous hands need to be coaxed towards the correct part of the observation space to make progress on a task. Thus, various approaches have used demonstrations to bootstrap the RL process [ 52 , 17 , 42 , 16 , 51 ] . In dexterous manipulation, this is often done through trajectory tracking, where instead of designing a complex reward function, a policy is rewarded for tracking the exact wrist and object positions in a demonstration, leaving RL to only make adjustments [ 6 , 75 ] . By re-framing dexterous manipulation as a trajectory tracking problem, such approaches can leverage dense, task-agnostic rewards and guide exploration by using residual policies [ 22 , 13 ] . Figure 1: Overview of our method: a VLM generates hand and object keypoint trajectories from a language command and scene image. A low-level residual RL policy is trained to track these trajectories in simulation. Though demonstration tracking overcomes the design challenges associated with RL, it paradoxically re-introduces the same dependence on demonstrations we sought to avoid in the first place. For example, prior works [ 6 , 75 ] that use tracking-based RL for dexterous manipulation often require large prior datasets with thousands of teleoperated demonstrations [ 11 , 72 ] , restricting the method to tasks for which data has already been collected. Furthermore, the dependence on accurate reference trajectories extends to test time, meaning that either new demonstrations are needed each time the environment changes, or a high-level policy must accurately predict them. In the context of prior work, either solution requires more demonstrations. However, for the majority of practical tasks it is unclear why demonstrations were chosen as reference trajectories when RL only requires that motions are accurate enough to provide a suitable reward and guide exploration. Our key insight is that coarse motion plans (“scaffolds”) can be sufficient for both of these criteria. Generating such plans only requires high-level spatial and semantic reasoning, the exact abilities afforded by new advancements in vision-language models (VLMs). Due to their training on vast and diverse datasets encompassing real-world knowledge and embodied reasoning [ 60 ] , VLMs have been able to synthesize [ 12 , 40 ] and discriminate [ 44 , 20 ] desired robot motions. Consequently, VLMs have the potential to supply the high-level reward signals and exploratory guidance needed for RL through coarse motions. So long as these motions generally encapsulate the desired behavior, RL can optimize per-timestep offsets and finger motions to maximize the tracking reward, ultimately surpassing human teleoperation in both performance and precision, eliminating the reliance on demonstrations. Building upon this insight, we introduce a framework for learning manipulation policies for dexterous robot hands with VLM-generated motion plans and residual RL. Given a natural language instruction (e.g., “hammer once” Fig. 1 ) and image, an off-the-shelf VLM first identifies relevant object keypoints. Then, provided the initial keypoints and hand pose, the VLM generates the associated 3D trajectories for both object and hand motions to define the supervision targets for a “low-level” residual RL policy trained in simulation. By controlling the robot’s hands and fingers, the low-level policy learns to effectively track the trajectory and complete the desired task. Using VLMs for high-level guidance has a number of distinct benefits. Through repeated querying, we can randomize the initial keypoints and high-level trajectories, enabling generalization to new unseen initial conditions and new trajectories at test-time. Moreover, in situations where a VLM’s high-level planning is error-prone, performance can often be substantially improved by simply providing additional in-context examples. We evaluate our method across a suite of challenging dexterous manipulation tasks in simulation requiring semantic understanding, human knowledge about concepts like “hammering”, and precise manipulation for difficult or articulated objects. Across 8 tasks, our method achieves close performance in both success rate and generalization to handcrafted, oracle plans despite requiring no manual reward engineering. We demonstrate successful sim-to-real transfer on a physical dexterous robot, achieving robust performance without any human demonstrations or manually designed rewards. 2 Related Work Planning with Vision-Language Models. Recent advances have demonstrated the potential of VLMs to guide robotic planning through their powerful semantic and

Read the complete text