Sensorized gripper for human demonstrations
Sri Harsha Turlapati, Gautami Golani, Mohammad Zaidi Ariffin, Domenico Campolo · 2025
Read Sensorized gripper for human demonstrations by Sri Harsha Turlapati, Gautami Golani, Mohammad Zaidi Ariffin, Domenico Campolo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14855.txt.
Introduction
Sri Harsha Turlapati, Gautami Golani, Mohammad Zaidi Ariffin, and Domenico Campolo build a sensorized gripper from off-the-shelf parts so robots can learn assembly by watching human hands. Their paper from twenty twenty-five pursues easy programming for messy environments. The gripper records a person performing a box-in-box assembly. Very few, very short demonstrations suffice for the robot to repeat the task.
Motion generation works in Cartesian space, solving joint positions while maximizing the arm's manipulability. Gaussian mixture models distill the demonstration statistics into a reusable skill. Impedance control then drives the robot through its copy of the task. Teaching robots by showing, not coding, is the long game this hardware serves.
Catalog record
| CREATOR | Sri Harsha Turlapati, Gautami Golani, Mohammad Zaidi Ariffin, Domenico Campolo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14855
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sensorized gripper for human demonstrations Sri Harsha Turlapati, Gautami Golani, Mohammad Zaidi Ariffin, Domenico Campolo Abstract Ease of programming is a key factor in making robots ubiquitous in unstructured environments. In this work, we present a sensorized gripper built with off-the-shelf parts, used to record human demonstrations of a box in box assembly task. With very few trials of short interval timings each, we show that a robot can repeat the task successfully. We adopt a Cartesian approach to robot motion generation by computing the joint space solution while concurrently solving for the optimal robot position, to maximise manipulability. The statistics of the human demonstration are extracted using Gaussian Mixture Models (GMM) and the robot is commanded using impedance control. keywords Learning from Demonstration, Impedance Control, Sensorized tools 1 Towards Intuitive Robot Programming The choice of deploying learning from demonstration (LfD) approaches Calinon2016 is well motivated by the need to make robot programming easy and accessible. Conventional choices in LfD include (i) directly recording humans by motion capture Artemiadis2010 (ii) Kinesthetic teaching Kormushev2011 and (iii) immersive teleoperation Kana2023 . Both teleoperation and kinesthetic teaching can be expensive and need to be safe for human interaction. The morphological differences Bicchi2000 between humans and robots make it hard to record human motion directly, e.g., a human hand with fingers is hardly comparable to a robot parallel jaw gripper. Anthropomorphic machine hands via sensorized gloves address this Perret2018 ; Almeida2019 . In this work, we propose instead to provide the human with a sensorized gripper (Fig. 1 ) to leverage on the existing skills and dexterity of humans. (a) Tool-perspective learning (b) Sensorized tool design Figure 1 : a) The tool is the only common aspect in the task. b) We choose off-the-shelf gripper and bicycle brake handle for the actuator lever to assemble the tool. In recent years, there has been an increasing trend in developing hand-held sensorized grippers Song2020 - Chi2024 . Vision based grasping datasets Song2020 - Shafiullah2023 have been produced with focus on manipulation policies obtained from object-centered observations. Similar to these works, we derive a Cartesian description of the task. Although visuo-tactile sensing Suh2022 was recently incorporated in the universal manipulation interface Chi2024 , to the best of authors’ knowledge, wrench sensing is missing in hand-held sensorized tool literature. Task wrenches can help the robot to adapt its motion in the case of failures, especially in the case of occlusions and visual uncertainty. In this work, we compare the haptics of human demonstration with the robot replay of the same task, and identify the possibility of robot trajectory adaptation as a future scope. For this reason, we choose to use impedance control to drive the robot, as opposed to standard position control. The scope of this work is only rigid body kinematics for now, in order to methodically develop this sensorized gripper platform. Once demonstration statistics are extracted, the robot will be driven at all joints. This requires the resolving of joint redundancy both for the robot joint space and its base positioning. The main goal is to quantify the success rate during the task repetition by the robot and the difference between task forces recorded during demonstration and actual robot replay, i.e., the haptic mismatch . 2 Cartesian approach to robot motion generation A robot’s coordinates usually are defined w.r.t the joint angles {\boldsymbol{q}}\in\mathbb{R}^{D} , while the task variables live in a Cartesian space. This poses the classical problem of determining robot kinematics by (i) computing joint space solution and (ii) determining an optimal location for the robot base. The tool in this paper is a gripper, whose pose is defined as ({\boldsymbol{R}}_{G},\ {\boldsymbol{p}}_{G})\in SE(3) . The distance between the gripper fingers is d_{G}\in\mathbb{R}^{+} . (a) Optimising robot base (b) Joint space solution (c) Computing manipulability Figure 2 : a) For robot base frame scenario \{s\} varying in a plane, robot manipulability is evaluated. b) Robot joint state is computed from gripper marker data using first order dynamics. c) The optimal robot base frame is determined after optimising the integral of manipulability across all scenarios. The sensorized gripper motion is recorded by the motion capture system, as a time indexed data-point {\boldsymbol{m}}_{i}(t) for marker i . Using SE(3) filtering GiaHoang2020 , we compute task space position {\boldsymbol{p}}_{G}(t) and orientation {\boldsymbol{R}}_{G}(t)\in SO(3) of the gripper frame from this marker data. The task space command {\boldsymbol{\Pi}} drives the robot gripper pose {\boldsymbol{T}} (computed using the forward kinematics block FK ) to track the demonstrated sensorized gripper pose ({\boldsymbol{p}}_{G},{\boldsymbol{R}}_{G}) with an initial configuration {\boldsymbol{q}}_{0} and the joint torque command {\boldsymbol{\tau}}_{PMP} . We implement first order system dynamics to compute the robot state ({\boldsymbol{q}},\dot{{\boldsymbol{q}}}) as outputs (Fig. 2b ). Thus, there are two components in {\boldsymbol{\Pi}} , the linear error and the rotational error with their corresponding stiffness k_{l} and k_{r} : {\boldsymbol{\Pi}}=\begin{bmatrix}k_{l}{\boldsymbol{R}}^{T}({\boldsymbol{p}}_{G}-{\boldsymbol{p}})\\ k_{r}\log_{\vee}({\boldsymbol{R}}^{T}{\boldsymbol{R}}_{G})\end{bmatrix} (1) where, \begin{bmatrix}{\boldsymbol{R}}&{\boldsymbol{p}}\\ {\boldsymbol{0}}^{T}&1\end{bmatrix}=FK({\boldsymbol{q}}) and \log_{\vee} is the SO(3) logarithm. We choose the cost function h , as one of the possible distances from robot singularity Chiaverini2008 with the aim of maximising manipulability during the robot motion: h=-\text{det}({\boldsymbol{J}}{\boldsymbol{J}}^{T}) (2) Using the identity for the derivative of a matrix determinant in Petersen2008 , we can compute the gradient \nabla_{{\boldsymbol{q}}}h of the cost as: \nabla_{{\boldsymbol{q}}}h=\left[\frac{\partial h}{\partial q_{1}}\dots\frac{\partial h}{\partial q_{7}}\right]^{T} (3) The individual gradient terms are computed using: \frac{\partial h}{\partial q_{i}}=-\text{det}({\boldsymbol{J}}{\boldsymbol{J}}^{T})\cdot\text{tr}\left(({\boldsymbol{J}}{\boldsymbol{J}}^{T})^{-1}\left(\frac{\partial{\boldsymbol{J}}}{\partial q_{i}}{\boldsymbol{J}}^{T}+{\boldsymbol{J}}\frac{\partial{\boldsymbol{J}}^{T}}{\partial q_{i}}\right)\right) 2.1 Determining optimal robot location Given a sensorized tool demonstration, it is not trivial to identify the robot base location, since the resultant robot joint space configurations for tool trajectory tracking, might not be optimal in terms of the robot’s manipulability. We optimise manipulability and compute it as a functional of the gripper trajectory given by marker data ({\boldsymbol{p}}_{G}(t),{\boldsymbol{R}}_{G}(t)) (Fig. 2 ). For each scenario s , we compute the robot joint trajectory {\boldsymbol{q}}_{s}(t)=[q_{1,s}\dots q_{7,s}]^{T} with the PMP block forward kinematics adjusted as if the robot is situated at frame \{s\} : \begin{bmatrix}{\boldsymbol{R}}&{\boldsymbol{p}}\\ {\boldsymbol{0}}_{1\times 3}&1\end{bmatrix}={\boldsymbol{T}}_{s}FK({\boldsymbol{q}}) (4) The cost integral, as an average manipulability across the entire volume of the gripper trajectory is computed as: \displaystyle\bar{\mu} \displaystyle=\frac{1}{t_{1}-t_{0}}\int_{t_{0}}^{t_{1}}\text{det}(J({\boldsymbol{q}}(t))J^{T}({\boldsymbol{q}}(t)))\,dt. (5) where, J is the robot Jacobian. The robot base is varied in position with limits x\in[x_{l}\dots x_{u}] , y\in[y_{l}\dots y_{u}] , and for every scenario s , we compute the robot manipulability \mu_{s} . The optimal robot base pose \{s^{*}\} is then selected which maximises this quantity. 3 From human demonstrations to robot replay In this work, we used the sensorized gripper to perform a square peg-in-hole task. Kinematically, the two frames that describe the task completely are (i) the gripper frame \{G\} and (ii) the box frame \{B\} . The demonstrations were performed by a human operator using a metronome to pace oneself to allow for consistent motions across trials, to capture some repeatability across time. The gripper frame information ({\boldsymbol{p}}_{G}(t_{j}),{\boldsymbol{R}}_{G}(t_{j})) for each trial j was computed using the SE(3) filter detailed in GiaHoang2020 . Additionally, two markers were placed on the gripper fingers to measure the distance d_{G}(t_{j}) between them to indicate when the gripper was closed and opened by the human operator. After repeated trials, this information was used to perform Gaussian Mixture Regression to compute the average demonstration trajectory in the task space to command the robot to follow. 3.1 Gaussian Mixture Regression (GMR) of demonstration