Variable-Friction In-Hand Manipulation for Arbitrary Objects via Diffusion-Based Imitation Learning
Qiyang Yan, Zihan Ding, Xin Zhou, Adam J. Spiers · 2025
Read Variable-Friction In-Hand Manipulation for Arbitrary Objects via Diffusion-Based Imitation Learning by Qiyang Yan, Zihan Ding, Xin Zhou, Adam J. Spiers (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02738.txt.
Introduction
Qiyang Yan and co-authors teach a robot hand new skills in a 2025 paper. Moving an object inside the hand to any pose is hard because contact is rich and subtle. A two-finger hand with changeable grip friction offers a simpler path, but past control code was hand-built for each shape.
Their method learns from examples, mixing simulated practice with a small set of real trials. Within two hours of training and one hour of real data, the hand could place mixed shapes, both regular and odd, at any target pose. Success topped seventy percent with tiny pose errors. Readers get a flexible route to deft handling without custom code per object.
Catalog record
| CREATOR | Qiyang Yan, Zihan Ding, Xin Zhou, Adam J. Spiers |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-02738
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Variable-Friction In-Hand Manipulation for Arbitrary Objects via Diffusion-Based Imitation Learning Qiyang Yan, Zihan Ding, Xin Zhou, Adam J. Spiers Abstract Dexterous in-hand manipulation (IHM) for arbitrary objects is challenging due to the rich and subtle contact process. Variable-friction manipulation is an alternative approach to dexterity, previously demonstrating robust and versatile 2D IHM capabilities with only two single-joint fingers. However, the hard-coded manipulation methods for variable friction hands are restricted to regular polygon objects and limited target poses, as well as requiring the policy to be tailored for each object. This paper proposes an end-to-end learning-based manipulation method to achieve arbitrary object manipulation for any target pose on real hardware, with minimal engineering efforts and data collection. The method features a diffusion policy-based imitation learning method with co-training from simulation and a small amount of real-world data. With the proposed framework, arbitrary objects including polygons and non-polygons can be precisely manipulated to reach arbitrary goal poses within 2 hours of training on an A100 GPU and only 1 hour of real-world data collection. The precision is higher than previous customized object-specific policies, achieving an average success rate of 71.3% with average pose error being 2.676 mm and 1.902 ∘ . Code and videos can be found at: https://sites.google.com/view/vf-ihm-il/home . I INTRODUCTION Humans effortlessly perform in-hand manipulation (IHM) for everyday tasks, such as reorienting and repositioning arbitrary complex objects like pens or keys without re-grasping [ 1 ] . These actions involve controlled transitions and re-orientations that are crucial but challenging for robots. While impressive IHM tasks have been performed by dexterous hands through learning-based methods [ 2 , 3 , 4 , 5 ] , those platforms tend to suffer from mechanical complexity. This complexity translates to extremely high hardware and maintenance costs as well as control complexity. Fig. 1: The manipulation trajectory of a complex Three-Cylinder object using the learned control policy. An Aruco marker obscures the body of the object, which is why the shape has been superimposed. In comparison, the 2-DOF variable-friction (VF) hand of [ 6 ] can achieve high object-dexterity despite its simple morphology. Inspired by the ability of human fingertips to selectively slide and grip objects, the VF-hand can achieve the same by actively switching its finger surface between high and low friction states. This allows objects to be slid or rolled along the finger surfaces. Fig. 2: In-hand manipulation of complex objects using the variable-friction hand, trained through our diffusion-based co-train imitation learning pipeline. Several manipulation-planning strategies have been developed for VF grippers [ 7 , 8 ] . The current state-of-the-art is a hybrid of planning and visual servoing [ 7 ] via an offline A* pathfinding algorithm to determine the most efficient trajectory through the combination of hard-coded model-based manipulation ‘primitives’. While this technique demonstrates the potential of variable friction hands, it also suffers from several limitations: (a) The model-based controller restricts manipulation to simple polygonal objects, such as cube and hexagonal prisms. (b) Rotational primitives only allow rolling an object from one flat surface to another. (c) Target poses (orientation and location) are restricted to positions where a flat surface of the object is in contact with a finger. We address these shortcomings via a new learning-based approach for VF-manipulation. An end-to-end data-efficient learning framework has been developed, based on imitation learning with diffusion policy. This enables the VF gripper to manipulate complex object shapes from arbitrary initial poses to arbitrary target poses in a continuous manipulation range. While our work focuses on the VF gripper, we believe this methodology is well suited to other uncommon gripper morphologies with non-holonomic object motion constraints. In addition, we determined that imitation learning with diffusion policy [ 9 ] can effectively learn high-precision policies, even with demonstrations generated via an imprecise RL policy optimized for manipulation smoothness. Finally, we have demonstrated that co-training the diffusion policy with a mix of real and simulated data increases both task success rate and resource efficiency. II Related Work Reinforcement learning (RL) learns optimal policies through interactions, and has shown promising results for in-hand manipulation (IHM) in recent years. Some work utilizing model-based approaches with explicit environment representations [ 10 , 11 , 12 ] , while model-free approaches learn policies directly [ 13 , 2 , 14 ] . IHM RL policy training is commonly performed in simulation [ 15 ] , as collecting enough interaction samples in the real world is often too resource intensive. However, training in simulation still requires a large amount of computing power (usually performed on CPU / GPU clusters) and training time due to IHM’s task complexity. Imitation Learning (IL) is a frequently-used method to reduce the required training effort for learned IHM policies [ 16 ] with growing popularity for IHM applications [ 17 ] . In addition, the behavior cloning in IL does not require carefully engineered simulation environments (no reward function is needed and it is more forgiving regarding action & observation space). Robot hands trained via this method learn from expert demonstrations, often generated via observing human object interactions [ 18 , 19 , 4 , 20 ] , teleoperation systems [ 21 , 22 ] , RL policies [ 3 ] , or hand-crafted control strategies [ 23 ] . Recently, IL with the diffusion policy [ 9 , 24 ] (which generates manipulation behaviour from demonstrations via a denoising diffusion process) has shown to be very effective. General tasks such as object-transportation or reorientation [ 9 ] , and even complex domestic tasks such as cooking or chair rearranging [ 25 ] have been successfully tested with this approach. In the context of IHM, [ 5 ] presented a complex hand with 4 fingers, each consisting of a delta mechanism, which used the diffusion policy to perform basic tasks such as cap twisting and syringe pushing. III Problem Formulation III-A Imitation Learning with Diffusion Policy The manipulation policy is formulated as a Denoising Diffusion Probabilistic Model (DDPM) [ 26 ] , which effectively captures the multi-modal distribution through a denoising process. For matching data distribution with probability p(x_{0}) , the forward diffusion process iteratively adds Gaussian noise \epsilon to the data sample such that at step k , we have x_{k}=\sqrt{\bar{\alpha}_{k}}x_{0}+\sqrt{1-\bar{\alpha}_{k}}\epsilon,\quad 1\leq k\leq K with noise \epsilon\sim\mathcal{N}(0,\mathbf{I}) and \bar{\alpha}_{k}=\Pi_{i=1}^{k}\alpha_{i} being a predetermined noise schedule. The reverse denoising process follows: \displaystyle x_{k-1}=\frac{1}{\sqrt{\alpha_{k}}}x_{k}-\frac{1-\alpha_{k}}{\sqrt{1-\bar{\alpha}_{k}}\sqrt{\alpha_{k}}}\epsilon_{\theta}+\mathcal{N}(0,\sigma^{2}_{k}I) (1) as the posterior distribution of Gaussian following the Bayes’ rule, with variance \sigma_{k}=\frac{(1-\alpha_{k})(1-\bar{\alpha}_{k-1})}{1-\bar{\alpha}_{k}} . \epsilon_{\theta} is the predicted noise by a model parameterized by \theta , which can be optimized with the following diffusion loss: \displaystyle\mathcal{L}=\mathbb{E}_{x_{0}\sim\mathcal{D},\epsilon\sim\mathcal{N}(0,\mathbf{I}),k\in\{1,\dots,K\}}(\epsilon-\epsilon_{\theta}(\sqrt{\bar{\alpha}_{k}}x_{0}+\sqrt{1-\bar{\alpha}_{k}}\epsilon,k))^{2} (2) The diffusion policy use DDPM to approximate the conditional distribution p(a|o) , i.e. , predicting action a conditioned on observation o . For imitation learning, the dataset \mathcal{D}=\{(o,a)\} is pre-collected with a given behavior policy \pi^{b} , and the current diffusion policy \pi_{\theta}(a|o) is optimized to minimize the distribution divergence from the behavior policy by Eq. ( 2 ). III-B Reinforcement Learning For RL, we define a Markov decision process (\mathcal{O},\mathcal{A},R,\mathcal{T},\gamma) , where \mathcal{O} is the observation space, \mathcal{A} is the action space, R(o,a):\mathcal{O}\times\mathcal{A}\rightarrow\mathbb{R} is the reward function, \mathcal{T}(o^{\prime}|o,a):\mathcal{O}\times\mathcal{A}\rightarrow\Pr(\mathcal{O}) is the stochastic transition function, and \gamma\in[0,1] is the discount factor for value estimation. The policy in