Functional Force-Aware Retargeting from Virtual Human Demos to Soft Robot Policies
Uksang Yoo 1 2 , Mengjia Zhu 2 , Evan Pezent 2 , Jom Preechayasomboon 2 , Jean Oh 1 , Jeffrey Ichnowski 1 , Amir Memar 2 , Ben Abbatematteo 2 , Homanga Bharadhwaj 2 , Ashish Deshpande 2 , Harsha Prahlad 2 · 2026
Read Functional Force-Aware Retargeting from Virtual Human Demos to Soft Robot Policies by Uksang Yoo 1 2 , Mengjia Zhu 2 , Evan Pezent 2 , Jom Preechayasomboon 2 , Jean Oh 1 , Jeffrey Ichnowski 1 , Amir Memar 2 , Ben Abbatematteo 2 , Homanga Bharadhwaj 2 , Ashish Deshpande 2 , Harsha Prahlad 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01224.txt.
Introduction
Yoo and colleagues introduce SoftAct, a framework that teaches soft robot hands human-like manipulation by reasoning explicitly about contact forces. Immersive virtual reality captures human demos, including hand motion, object motion, dense contact patches, and detailed force data. A two-stage retargeting algorithm first assigns demo forces to human fingers and allocates robot fingers in proportion, then refines fingertip targets online using contact geometry and force size. Tests on a custom pneumatic soft hand show fingertip tracking error falls by up to 55 percent.
Catalog record
| CREATOR | Uksang Yoo 1 2 , Mengjia Zhu 2 , Evan Pezent 2 , Jom Preechayasomboon 2 , Jean Oh 1 , Jeffrey Ichnowski 1 , Amir Memar 2 , Ben Abbatematteo 2 , Homanga Bharadhwaj 2 , Ashish Deshpande 2 , Harsha Prahlad 2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49781 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.01224
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Functional Force-Aware Retargeting from Virtual Human Demos to Soft Robot Policies Uksang Yoo 1 2 , Mengjia Zhu 2 , Evan Pezent 2 , Jom Preechayasomboon 2 , Jean Oh 1 , Jeffrey Ichnowski 1 , Amir Memar 2 , Ben Abbatematteo 2 , Homanga Bharadhwaj 2 , Ashish Deshpande 2 , Harsha Prahlad 2 Abstract We introduce SoftAct , a framework for teaching soft robot hands to perform human-like manipulation skills by explicitly reasoning about contact forces. Leveraging immersive virtual reality, our system captures rich human demonstrations, including hand kinematics, object motion, dense contact patches, and detailed contact force information. Unlike conventional approaches that retarget human joint trajectories, SoftAct employs a two-stage, force-aware retargeting algorithm. The first stage attributes demonstrated contact forces to individual human fingers and allocates robot fingers proportionally, establishing a force-balanced mapping between human and robot hands. The second stage performs online retargeting by combining baseline end-effector pose tracking with geodesic-weighted contact refinements, using contact geometry and force magnitude to adjust robot fingertip targets in real time. This formulation enables soft robotic hands to reproduce the functional intent of human demonstrations while naturally accommodating extreme embodiment mismatch and nonlinear compliance. We evaluate SoftAct on a suite of contact-rich manipulation tasks using a custom non-anthropomorphic pneumatic soft robot hand. SoftAct ’s controller reduces fingertip trajectory tracking RMSE by up to 55% and reduces tracking variance by up to 69% compared to kinematic and learning-based baselines. At the policy level, SoftAct achieves consistently higher success in zero-shot real-world deployment and in simulation. These results demonstrate that explicitly modeling contact geometry and force distribution is essential for effective skill transfer to soft robotic hands, and cannot be recovered through kinematic imitation alone. Project videos and additional details are available at soft-act.github.io . soft-act.github.io I Introduction Soft robot hands offer intrinsic compliance, safety in human interaction, and robustness to uncertainty, making them particularly well suited for contact-rich manipulation tasks [ 33 , 36 , 37 , 6 , 38 ] . These advantages, however, come with fundamental challenges for policy learning and control [ 26 ] . In particular, translating dexterous manipulation skills from human demonstrations to soft robotic hands remains difficult due to extreme differences in workspaces, affordances, and contrability. Unlike rigid and anthropomorphic robot hands, soft robotic hands are typically underactuated and driven with nonlinear actuators such as artificial muscles, exhibiting nonlinear, deformation-based motion rather than jointed kinematics [ 1 , 30 ] . Their morphologies and finger configurations often differ substantially from the human hand in degrees of freedom, workspace geometry, and contact affordances [ 3 ] . As a result, direct correspondence between human and robot configurations is ill-defined, rendering classical imitation and retargeting approaches ineffective for many soft-hand systems. Prior work on learning from demonstration and hand retargeting often assumes a meaningful kinematic mapping and finger-to-finger correspondences between the demonstrator and the robot [ 28 , 4 , 21 , 18 ] . Human joint trajectories, fingertip positions, or full hand poses are mapped to robot configurations via inverse kinematics, heuristic rules, or learned pose-to-pose mappings [ 7 , 24 , 25 , 16 ] . While effective for anthropomorphic robot hands with similar structure, these methods break down for non-anthropomorphic and soft hands as we demonstrate in our experiments, where kinematic matching is often ill-posed and insufficient to reproduce functional manipulation, even when approximate geometric similarity is achieved. Contact information can provide some guidance in generalizing manipulation strategies form human demonstrators to the robot embodiments [ 15 , 18 , 16 ] . However two limitations exist: obtaining such contact labeled datasets and these methods still require an explicit mapping between the hand surfaces to align the contact patches [ 15 ] , limiting their applications to mostly anthropomorphic target embodiments. In this paper, we propose SoftAct , a contact force-guided learning-from-demonstration framework for transferring human manipulation skills to non-anthropomorphic soft robotic hands. Our approach leverages immersive virtual reality to collect human demonstrations, capturing not only hand kinematics and object motion, but also dense contact patches and detailed contact force information throughout manipulation. Rather than directly retargeting human joint trajectories, SoftAct introduces a two-stage force-aware retargeting algorithm. In the first stage, contact forces are attributed to individual human fingers, and robot fingers are allocated proportionally, establishing a force-balanced mapping between the human and robot hands. In the second stage, online retargeting is performed by combining baseline end-effector pose tracking with geodesic-weighted fine adjustments, using contact and force data to refine robot fingertip targets in real time. This formulation enables the robot to reproduce the functional demonstration by explicitly optimizing for geometry in contact points, while respecting the unique kinematic and morphological constraints of soft robotic hands. By optimizing contact- and force-based objectives under the constraints of a pneumatically actuated soft hand, SoftAct naturally discovers robot-specific manipulation strategies that may differ substantially from human motion, yet achieve similar interaction outcomes. This stands in contrast to kinematics-driven retargeting methods, which enforce geometric similarity and often struggle under severe embodiment mismatch. We evaluate our approach on multiple contact-rich manipulation tasks using a custom non-anthropomorphic pneumatic soft robot hand. Across tasks, our contact-centric retargeting produces more stable interactions and higher task success rates than kinematics-only baselines. Ablation studies further highlight the contributions of individual system components. The resulting behaviors exhibit distinct contact patterns and force distributions that reflect the unique affordances of pneumatic actuation, demonstrating that contact-level representations are essential for transferring human manipulation skills to soft robotic hands. Finally, we validate the simulated results with zero-shot real-world deployment, showing that SoftAct produces more stable and reliable rollout performances. In summary, we make the following contributions: • A contact-centric learning-from-demonstration formulation for retargeting human manipulation skills to non-anthropomorphic soft robotic hands. • A force-aware retargeting method that explicitly optimizes contact patches and force distributions rather than hardcoded kinematic similarity. • A state-of-the-art learning-based low-level controller for tracking trajectories with the highly nonlinear pneumatic soft robot hand. • An end-to-end system for deploying the policy trained from virtual reality datasets on a pneumatically actuated soft hand, evaluated across multiple contact-rich tasks. II Related Work We discuss prior works in dexterous manipulation with soft hands, learning from expert demonstrations, and human to robot motion retargeting. II-A Dexterous Soft Robot Manipulation Soft robotic hands have been widely studied for manipulation due to their intrinsic compliance, which enables passive adaptation to object geometry, robustness to uncertainty, and safe interaction with humans and fragile objects [ 26 , 6 , 41 ] . By distributing contact forces over extended surface areas, soft hands can form stable grasps and tolerate positioning errors that would lead to failure in rigid, jointed hands [ 1 , 30 , 17 ] . Prior work has improved the dexterity of soft hands through advances in actuator design, material composition, and bio-inspired morphologies [ 23 , 10 ] . These developments have enabled increasingly sophisticated grasping behaviors and limited forms of in-hand manipulation, often relying on carefully hand-designed primitives and substantial expert tuning [ 23 ] . Despite this progress, controlling soft hands remains fundamentally challenging due to their nonlinear, underactuated dynamics and deformation-driven motion, which complicate modeling, state estimation, and control [ 34 ] . As a result,