Learning to Design Soft Hands using Reward Models
Xueqian Bai, Nicklas Hansen, Adabhav Singh, Michael T. Tolley, Yan Duan, Pieter Abbeel, Xiaolong Wang, Sha Yi · 2025
Read Learning to Design Soft Hands using Reward Models by Xueqian Bai, Nicklas Hansen, Adabhav Singh, Michael T. Tolley, Yan Duan, Pieter Abbeel, Xiaolong Wang, Sha Yi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17086.txt.
Introduction
Xueqian Bai and seven co-authors tackle the design of soft robot hands in this 2025 paper. Soft hands bend and grip with care, but good shapes are hard to find by hand. Testing each shape in a simulator is slow and costly. The team wants a faster way to search for strong designs.
Their method learns a reward model from past remote-control trials. A search rule then tests far fewer shapes than brute force would need. The best shapes are printed in three dimensions and tried on real tasks. The printed hands beat standard hands at grasping hard objects. The work cuts design time while keeping hands gentle and useful.
Catalog record
| CREATOR | Xueqian Bai, Nicklas Hansen, Adabhav Singh, Michael T. Tolley, Yan Duan, Pieter Abbeel, Xiaolong Wang, Sha Yi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17086
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Design Soft Hands using Reward Models Xueqian Bai, Nicklas Hansen, Adabhav Singh, Michael T. Tolley, Yan Duan, Pieter Abbeel, Xiaolong Wang, Sha Yi Abstract Soft robotic hands promise to provide compliant and safe interaction with objects and environments. However, designing soft hands to be both compliant and functional across diverse use cases remains challenging. Although co-design of hardware and control better couples morphology to behavior [ 1 ] , the resulting search space is high-dimensional, and even simulation-based evaluation is computationally expensive. In this paper, we propose a Cross-Entropy Method with Reward Model (CEM-RM) framework that efficiently optimizes tendon-driven soft robotic hands based on teleoperation control policy, reducing design evaluations by more than half compared to pure optimization while learning a distribution of optimized hand designs from pre-collected teleoperation data. We derive a design space for a soft robotic hand composed of flexural soft fingers and implement parallelized training in simulation. The optimized hands are then 3D-printed and deployed in the real world using both teleoperation data and real-time teleoperation. Experiments in both simulation and hardware demonstrate that our optimized design significantly outperforms baseline hands in grasping success rates across a diverse set of challenging objects. 1 UC San Diego, 2 Amazon FAR (Frontier AI & Robotics) Fig. 1 : We present a Cross-Entropy Method (CEM) with reward model ( CEM-RM ) framework that optimizes block-wise, finger-wise, and tendon-routing design distributions of a soft robotic hand using pre-collected teleoperation data. Hardware experiments demonstrate that CEM-RM achieves effective design optimization with significantly fewer samples than pure optimization, enabling robust grasping of challenging objects. I Introduction Recent progress in teleoperation [ 2 , 3 ] and rapid fabrication technologies [ 4 ] have created powerful opportunities for developing and evaluating novel robotic hand designs. Many of these are done with rigid robotic hands, which are straightforward to model and control [ 5 ] . However, rigid robotic hands remain costly as they require small and accurate motors, and tend to be fragile when executing contact-rich interactions in a tabletop setting. The 3D printers have become accessible and greatly enabled rapid prototyping of alternative hand morphologies, including soft hands. Soft hands are inherently safe to work around humans, objects, and environments. Soft fingers actuated with tendons [ 6 ] are especially easy to make: they can be directly 3D-printed in a single piece [ 7 , 8 ] with integrated flexure joints. This allows us to quickly explore a broad design space by validating the hardware design in the real world. Advances in computational methods and machine learning are transforming how robotic design and control can be optimized. Gradient- and planning-based methods have been explored to optimize the geometry of robots [ 9 , 10 ] . However, these methods typically require extensive domain expertise. Machine learning offers a data-driven alternative [ 11 , 12 ] , yet design problems are still challenging because they lack direct supervision: engineers often rely on intuition or costly trial-and-error iterations. This is particularly limiting when designs must generalize across different use cases. Co-design of both control and design has been shown to effectively connect designs with their use cases [ 13 , 8 ] . However, design and control by themselves are hard problems with high dimensionality. The joint co-design space can quickly become intractable. In this work, we present a framework for learning to design soft robotic hands using a reward model. Our approach uses control demonstrations from teleoperation collected on real objects and transfers them into simulation. We sample a wide range of hand design parameters and simulate the corresponding grasping outcomes. To explore the design space efficiently, we employ the Cross-Entropy Method (CEM), which maintains and iteratively refines a sampling distribution over design parameters by resampling from the top-performing candidates. However, this process can still be slow and costly. Thus, we use the simulation grasping outcomes as ground-truth rewards to train a neural reward model based on a multilayer perceptron (MLP). By combining the reward model with CEM, we significantly accelerate the design search: the reward model reduces the required simulation budget by half while maintaining its fidelity. We then validate the optimized designs in both simulation and hardware by fabricating hands via 3D printing. Results show that our learned designs outperform a uniform baseline, achieving higher grasp success rates. Overall, our approach demonstrates how to efficiently explore large design space using sampling and reward models. More broadly, it offers an option for generating optimal robot designs given an existing control policy, without requiring domain-specific analytic models or excessive simulation trials. II Related Work Robotic hand design faces challenges from high-dimensional parameter spaces and the complexity of optimizing control policies. Recent advances in machine learning and world models have improved sample efficiency, while teleoperation offers reliable control demonstrations that bypass the need to learn policies entirely from scratch. Computational Design and Co-Design. Computational methods use optimization algorithms, simulation, and data analysis to enhance the design process. Prior work has explored planning [ 14 ] and gradient-based optimization over geometry and topology [ 15 ] . However, such approaches often fail to generalize across tasks and struggle with discontinuous contacts or fabrication constraints that make gradients intractable. Bi-level formulations jointly optimizing design and control [ 16 , 17 ] alleviate this, while simulators [ 18 ] enable end-to-end co-design but still remain challenging with non-linearity and sensitive to initialization [ 13 , 19 ] . Sampling-based frameworks instead trade computational cost for broader exploration: Genetic Algorithms [ 20 ] and Covariance Matrix Adaptation - Evolution Strategy (CMA-ES) [ 21 ] evaluate large populations in discrete or continuous spaces; Bayesian Optimization balances exploration and exploitation in low-dimensional spaces [ 22 ] ; and Reinforcement Learning (RL) coupled with evolutionary methods discovers unintuitive morphologies [ 23 , 1 , 24 ] . Tendon-driven Robotic Hands. Tendons are a popular way of actuating robotics hands [ 6 ] . Compared to direct-drive hands such as LEAP [ 25 ] , tendon-driven hands [ 26 , 27 , 28 , 29 ] take inspiration from biological hands to control joints with tendons driven by actuators positioned away from fingers. Some soft designs encode joints directly into monolithic bodies [ 7 , 30 ] or exploit variable-stiffness materials for adaptable compliance [ 31 ] . However, designing a tendon-driven soft robot body still relies on manual tuning and the exploration of geometric parameters. Optimization with Function Approximation. Data-driven approaches to optimization such as Reinforcement Learning (RL) have flourished in the era of neural networks, and a variety of broadly applicable RL algorithms have been proposed. These algorithms typically have an “actor-critic” structure in which a critic approximates the (usually temporally extended) problem landscape, and an actor concurrently learns to maximize expected return as estimated by the critic [ 32 , 33 , 34 , 35 ] . A key benefit of such methods is their ability to model high-dimensional and often highly nonlinear problem landscapes. Most recently, researchers find that such algorithms can be simplified by replacing the actor with a non-parametric function such as the Cross-Entropy Method (CEM) [ 36 , 37 , 38 , 39 , 40 , 41 ] , or the critic with