DexTwist: Dexterous Hand Retargeting for Twist Motion via Mixed Reality-based Teleoperation

Dongmyoung Lee, Chengxi Li, Dongheui Lee · 2026

Read DexTwist: Dexterous Hand Retargeting for Twist Motion via Mixed Reality-based Teleoperation by Dongmyoung Lee, Chengxi Li, Dongheui Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12182.txt.

Introduction

Dongmyoung Lee, Chengxi Li, and Dongheui Lee present DexTwist, twist retargeting for MR control of dexterous hands. Standard retargeting copies joint angles or fingertips, but body mismatches slide fingertips sideways during twists. Cap opening, key turning, and bolt screwing then drift the screw axis and loosen the grasp. DexTwist spots a tripod pinch, guesses the screw axis and twist size, refines joints to track turning while steadying the tripod shape. Simulated and real tests show better turning-angle tracking and screw-axis stability than a vector baseline.

Catalog record

CREATORDongmyoung Lee, Chengxi Li, Dongheui Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28472
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12182

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DexTwist: Dexterous Hand Retargeting for Twist Motion via Mixed Reality-based Teleoperation Dongmyoung Lee, Chengxi Li, Dongheui Lee Abstract Dexterous teleoperation via Mixed Reality (MR)-based interfaces offers a scalable paradigm for transferring human manipulation skills to dexterous robot hands. However, conventional retargeting approaches that minimize kinematic dissimilarity (e.g., joint angle or fingertip position error) often fail in contact-rich rotational manipulation, such as cap opening, key turning, and bolt screwing. This failure stems from the embodiment gap: mismatched link lengths, joint axes/limits, and fingertip geometry can cause direct pose imitation to induce tangential fingertip sliding rather than stable object rotation, resulting in screw axis drift, contact slip, and grasp instability. To address this, we propose DexTwist, a functional twist-retargeting framework for MR-based dexterous teleoperation. DexTwist detects a tripod pinch, estimates the operator’s intended screw axis and twist magnitude, and applies a real-time residual joint-space refinement that tracks turning progress while regularizing the robot tripod geometry. The refinement minimizes a virtual-object objective defined by turning angle, screw axis consistency, fingertip closure, and tripod stability. Simulation and real-world experiments show that DexTwist improves turning angle tracking and screw axis stability compared with a vector-based retargeting baseline. I Introduction Dexterous robotic hands offer the promise of human-level manipulation capability, but reliable dexterous control remains challenging because contact-rich tasks require precise coordination of high-dimensional end-effectors to interact with diverse objects. Recent advances in vision-based hand tracking allow operators to transfer manipulation skills to robotic hands without specialized master devices or motion-capture systems, lowering deployment barriers in real-world environments [ 8 , 18 , 5 ] . Meanwhile, building on these advances, Mixed-Reality Head-Mounted Display (MR-HMD) provides intuitive and mobile teleoperation interfaces for robot learning demonstrations and human fallback when autonomous execution fails [ 3 , 4 ] . Throughout this paper, MR-based teleoperation leverages a headset interface that combines egocentric hand tracking and streamed visual feedback from the robot workspace. This interface allows the operator to provide hand/wrist motion commands while monitoring the real-world manipulation scene through the headset, forming an integrated sensing and feedback platform for dexterous teleoperation. In terms of retargeting, despite high fidelity in free-space imitation, teleoperation often struggles in contact-rich interactions such as opening a cap, rotating a knob, or twisting a tool. A primary bottleneck is the embodiment gap: differences in kinematics and geometry between human and robotic hands mean that joint- or pose-level similarity does not necessarily produce correct contact behavior. Conventional retargeting objectives therefore work well for gesture imitation but provide weak guarantees for contact-rich manipulation [ 1 , 11 ] . For rotational tasks such as screwing, pose similarity is an insufficient proxy for task success. The robot should generate rotation about a physical axis while maintaining stable fingertip contacts and avoiding slip. Under embodiment mismatch, direct kinematic mapping can produce tangential fingertip motion along the object surface rather than rotation about the intended axis, leading to screw axis drift, contact slip, and loss of grasp stability. These failure modes motivate a shift from gesture imitation to function preservation. Figure 1: An overview of the proposed system DexTwist, a robot teleoperation and hand retargeting framework that supports achieving dexterous screw manipulation In this work, we propose DexTwist, an MR-based teleoperation system with functional hand retargeting for in-hand twist manipulation, as shown in Fig. 1 . DexTwist commands the robot arm, detects a human tripod pinch, estimates the screw axis and accumulated twist intent in the operator’s palm frame, and generates robot hand commands through real-time residual refinement. The objective tracks the intended turning angle while penalizing screw axis drift, loss of fingertip closure, and tripod centroid translation, preserving the turning function under human-robot embodiment mismatch. Reliable twist teleoperation is relevant to remote maintenance, inspection, disaster response, and industrial servicing, where operators may need to manipulate caps, keys, valves, or fasteners without direct physical presence. The contributions of this work are summarized as follows: 1. An MR-based dexterous teleoperation pipeline with robust tripod activation and episode-level turning state that supports ratcheting. 2. A functional twist retargeting module implemented as a residual joint-space refinement, minimizing a virtual-object objective defined in the robot palm frame to track turning progress while regularizing tripod stability. 3. An evaluation protocol comparing DexTwist with vector-based retargeting in simulation and real-world hardware. II Related Works II-A Robot Teleoperation Teleoperation has become a key paradigm for large-scale robotic manipulation data collection, evolving from parallel-jaw grippers [ 19 , 7 ] to anthropomorphic dexterous hands. While multi-fingered platforms enable richer manipulation capabilities, they substantially increase teleoperation complexity, leading prior work to explore exoskeletons [ 6 ] , motion capture systems [ 13 ] , MR devices [ 2 ] , and vision-based interfaces. Among these, MR-based approaches offer improved accessibility and immersion, yet existing systems largely focus on pose-level control and data acquisition, leaving functional manipulation intent and contact-aware execution underexplored. Lipton et al. [ 12 ] and Wan et al. [ 20 ] leverage off-the-shelf Virtual-Reality(VR) hardware to enable immersive human control, they achieve this by embedding the operator in a virtual control room mapped to the robot’s workspace while providing rich sensor feedback and dynamic human–robot mapping. Ponomareva et al. [ 17 ] proposed an augmented virtual environment for medical teleoperation that alleviates the limitations of camera-based visual feedback, such as restricted viewpoints and high cognitive load. Regarding the MR devices, Ding et al. introduce Bunny-VisionPro [ 4 ] , a real-time bimanual dexterous teleoperation system that leverages an MR headset, and further design novel low-cost devices to provide haptic feedback to the operator, enhancing immersion. Similarly, Cheng et al. [ 3 ] presented an immersive teleoperation system designed to provide stereoscopic perception and mirroring the operator’s arm and hand motions onto humanoid robots. Moreover, some recent works also focus on scalable, user-friendly teleoperation platforms to address the scarcity of high-quality robotic manipulation data. OpenTeach [ 9 ] enables low-cost, immersive MR-based teleoperation across diverse robot morphologies for collecting dexterous, contact-rich demonstrations, while DART [ 16 ] reimagines data collection through cloud-based simulation and MR devices, enabling crowdsourced, high-throughput data generation with reduced operator fatigue. Both systems demonstrate that data collected through intuitive teleoperation—either in real-world MR or cloud simulation—can effectively support policy learning and transfer to real robots. Previous research has highlighted the effectiveness of MR applications in the realm of teleoperation with ease. Currently, the predominant focus of utilizing MR-headset revolves around extracting human information for robot motion control, it still does not leverage the rich visualization feature well for providing information for human operator to visualization and monitoring. II-B Hand Motion Retargeting In dexterous hand teleoperation, accurately retargeting human hand motion to robotic hands remains challenging because human and robotic hands differ in morphology, kinematic structure, joint limits, fingertip geometry, and degrees of freedom. Prior work has addressed this problem using hand motion captured from motion-capture systems, VR headsets, data gloves, or vision-based tracking. Learning-based methods align human and robot hand motions through shared latent spaces or directly map visual hand observations to robot joint commands [ 21 , 11 ] . These approaches are attractive for scalable retargeting, but often require representative training data and tend to reproduce hand pose or joint configurations. Optimization-based methods instead formulate retargeting through fingertip positions, task-space

Read the complete text