Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation

Gabriele Giudici, Aramis Augusto Bonzini, Claudio Coppola, Kaspar Althoefer, Ildar Farkhatdinov, Lorenzo Jamone · 2024

Read Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation by Gabriele Giudici, Aramis Augusto Bonzini, Claudio Coppola, Kaspar Althoefer, Ildar Farkhatdinov, Lorenzo Jamone (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02644.txt.

Introduction

Gabriele Giudici and co-authors show that a robot arm can be steered by touch alone, without cameras. Remote manipulation usually leans on video, which fails when views are blocked or smoke and darkness ruin the image. Their system reads tactile sensors on the gripper, rebuilds the touched object as a 3D model inside a virtual reality scene, and feeds touch sensations back to the operator's hand. A blindfolded user completed precise pick-and-place tasks this way.

The authors present the result as a feasibility proof, not a finished product. Still, tasks that once depended on cameras were done by feel. That opens settings where cameras struggle, such as disaster sites or underwater work. The study also matters for training robot agents, where touch-based demonstrations could teach skills that vision alone cannot capture. Future work can add complexity beyond simple pick-and-place.

Catalog record

CREATORGabriele Giudici, Aramis Augusto Bonzini, Claudio Coppola, Kaspar Althoefer, Ildar Farkhatdinov, Lorenzo Jamone
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-02644

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation Gabriele Giudici, Aramis Augusto Bonzini, Claudio Coppola, Kaspar Althoefer, Ildar Farkhatdinov, Lorenzo Jamone Abstract Dexterous robotic manipulator teleoperation is widely used in many applications, either where it is convenient to keep the human inside the control loop, or to train advanced robot agents. So far, this technology has been used in combination with camera systems with remarkable success. On the other hand, only a limited number of studies have focused on leveraging haptic feedback from tactile sensors in contexts where camera-based systems fail, such as due to self-occlusions or poor light conditions like smoke. This study demonstrates the feasibility of precise pick-and-place teleoperation without cameras by leveraging tactile-based 3D object reconstruction in VR and providing haptic feedback to a blindfolded user. Our preliminary results show that integrating these technologies enables the successful completion of telemanipulation tasks previously dependent on cameras, paving the way for more complex future applications. I INTRODUCTION Robotic teleoperation enables the remote control of a robotic system by a human operator for tasks that are dangerous, inaccessible, or intricate [ 1 ] . This is required in numerous applications where current control techniques fall short in achieving autonomous robotic operations, where it is preferable to maintain a human in the loop, and where it is useful to transfer the human skills to the robot resorting to the learning by demonstration paradigm. Most teleoperation activities rely on vision-based systems, which, through cameras, allow manipulation experiments to be carried out at different levels of complexity [ 2 ] . Such systems are not infallible, and harsh visual conditions such as poor light, the presence of shadows, haze, smoke, radiation [ 3 ] or even dirty or damaged camera lenses limit their reliability [ 4 ] . Tactile sensing can address some of these issues as it can provide measurements of properties that cannot be easily estimated with vision alone, such as, weight or texture information of the manipulated object surface [ 5 , 6 ] ; for example, they can be used to reconstruct the 3D shape of an object [ 7 ] , [ 8 ] , which could then be displayed using VR or AR technologies. In addition, tactile sensing can be used to provide haptic and force feedback to the human operator, obtaining bilateral telerobotic systems capable of delivering an immersive haptic experience [ 9 ] , [ 10 ] . Fig. 1: View of the human-operator setup (haptic interface with the glove and a VR visor) and the teleoperated robotic hand. So far, to the best of our knowledge, no other work has demonstrated the feasibility of conducting dexterous telemanipulation experiments in real-world settings without the aid of cameras or human vision of the target’s real workspace. Therefore, to address the current research gap and the significant demands posed by teleoperation in complete blindness, we have integrated a visual Virtual Reality (VR) visor into our established telerobotics setup [ 11 ] . The adopted bilateral teleoperation system exclusively relies on tactile sensing from the robot hand, omitting visual sensors. This system delivers crucial haptic feedback to the human user and facilitates accurate object manipulation. Moreover, we have developed a minimalist virtual environment to visualize the robotic arm in real-time during 3D object reconstruction and manipulation tasks. Utilizing only tactile sensors and Gaussian Process (GP), our system streams real-time partial 3D reconstructions of the shape of the manipulated object onto the visor while rendering haptic experience of the manipulated object, enhancing immersion and reducing fatigue. This aim to mitigate the challenges associated with blind teleoperation, where it is challenging, if not impossible, for the teleoperator to manipulate objects accurately without haptic feedback alongside VR support. In this work, we provide experimental evidence that a human operator, with prior experience in using vision-supervised teleoperation, can perform blind pick-and-place tasks accurately supported exclusively by haptic feedback and the VR visual representation. To summarize, the main contributions of this work are: • Integration of Virtual Reality into the teleoperation setup from [ 11 ] , so that tactile sensing on the robot hand is used not only to render haptic feedback through an exoskeletal glove but also to generate a real-time 3D reconstruction of the object shape via a Meta Quest 2 headset, thus providing rich feedback to the operator without cameras. • A study demonstrating the feasibility of precise pick-and-place tasks using blind teleoperation, where the operator is isolated from the real scene and relies solely on tactile-based virtual object reconstruction and haptic feedback. The paper is organized as follows: Section II reviews related work. Section III outlines the telemanipulation system and VR elements. Section IV describes the experimental setup, and Section 5 details the results. In Section VI , we discuss the findings, and Section VII suggests future research directions. II RELATED WORK The system we propose is pioneering, as no real-world robotic telemanipulation system in the literature, to the best of our knowledge, avoids reliance on visual sensing. However, in this section, we briefly introduce related works in two key areas: autonomous robotic manipulation via tactile feedback alone (excluding visual cues) and multimodal telemanipulation. We considered it unnecessary to compare the performance and accuracy of our work with studies employing vision-based approaches, as our work is not intended to be an alternative but rather a complement to systems utilizing cameras. II-A Tactile Manipulation Although the interest in object manipulation is widely studied in the field of robotics [ 12 ] , [ 13 ] , [ 14 ] , the exploration of manipulating objects without reliance on a vision system remains a relatively less explored field. In [ 15 ] , the authors proposed a learning method based on a touch scan through which it was possible to identify the position of an object to grasp it stably. This work [ 16 ] introduced a tactile closed-loop method using deep reinforcement learning to improve grasping stability and overcome the limitations of vision-based robotic grasping systems prone to calibration errors and occlusion. In this study [ 17 ] , the authors presented a robotic system able to localize and grasp freely moving objects only using only information gathered via local touch sensors without any visual feedback. In [ 18 ] , a framework to segment human demonstrations is presented to support the learning of primitive skills that compose long-horizon tasks. Although this framework does not require the use of cameras, during the telemanipulation demonstrations the operator has a direct vision of the workspace. Different works have furthermore proven how it is possible through tactile information to reconstruct the 3D shape of an object [ 5 ] . Amongst the various approaches [ 19 ] , [ 20 ] , [ 8 ] which by using GP and point clouds were able to reconstruct the object with significant results. However, none of these methods have been integrated into a telerobotic setup that allows the human operator to manually control the robot and perceive a haptic experience of the manipulated object. II-B Multimodal Telemanipulation The significance of a haptic experience made possible through touch is thoroughly explored in this review [ 21

Read the complete text