ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface
Fangchen Liu, Chuanyu Li, Yihua Qin, Jing Xu, Pieter Abbeel, Rui Chen · 2025
Read ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface by Fangchen Liu, Chuanyu Li, Yihua Qin, Jing Xu, Pieter Abbeel, Rui Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06156.txt.
Introduction
Fangchen Liu and co-authors build a hand held gripper that lets people teach robots touch rich skills without operating a robot at all, in a paper from 2025. Robots learn delicate handling from demonstrations, but collecting them through teleoperation rigs is slow and costly. Touch, the sense that matters most in contact tasks, is also the hardest to record that way.
Their device, ViTaMIn, pairs a soft fin ray gripper with tactile sensors and a camera in one hand tool. A human simply performs the task; the gripper feels the forces and records them with the view. A pretraining scheme turns the touch stream into reusable features, so policies learn from fewer demonstrations. The interface aims to make robot teaching data as easy to gather as video.
Catalog record
| CREATOR | Fangchen Liu, Chuanyu Li, Yihua Qin, Jing Xu, Pieter Abbeel, Rui Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06156
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface Fangchen Liu, Chuanyu Li, Yihua Qin, Jing Xu, Pieter Abbeel, Rui Chen Abstract Tactile information plays a crucial role for humans and robots to interact effectively with their environment, particularly for tasks requiring the understanding of contact properties. Solving such dexterous manipulation tasks often relies on imitation learning from demonstration datasets, which are typically collected via teleoperation systems and often demand substantial time and effort. To address these challenges, we present ViTaMIn, an embodiment-free manipulation interface that integrates visual and tactile sensing into a hand-held gripper, enabling multi-modality data collection without the need for teleoperation. Our design employs a compliant Fin Ray gripper with tactile sensing, allowing operators to perceive force feedback during manipulation for more intuitive operation. Additionally, we propose a multi-modal representation learning strategy to obtain pre-trained tactile representations, improving data efficiency and policy robustness. Experiments on 5 contact-rich manipulation tasks demonstrate that our system is more scalable, efficient, and effective than baseline methods. I Introduction Humans rely on both visual and tactile modalities to perform a diverse range of manipulation tasks in daily life. For instance, when inserting a plug into a socket or tightening a screw, vision helps with identifying and aligning components, while tactile signals enable precise force control during contact. This seamless integration of vision and touch enhances human dexterity, particularly in tasks that require contact-rich control, handling visual occlusions, or performing in-hand manipulations. Recent progress in learning from demonstrations [ 1 , 2 , 3 , 4 ] has shown significant potential for advancing general-purpose robots, enabling them to efficiently acquire complex skills from human demonstrations. Consequently, developing systems to collect high-quality demonstration data has been a recent key focus. Prior works have explored real-world data collection methods, including joint-mapped devices and exoskeletons [ 5 , 6 , 7 , 8 ] , and vision-based teleoperation frameworks [ 9 , 10 ] . Nevertheless, these techniques require real-time teleoperation of a physical robot during data collection, which constrains efficiency and flexibility. In contrast, portable devices [ 11 , 12 , 13 , 14 ] present a more scalable and cost-effective alternative to collect demonstration without teleoperation. Moreover, they can be seamlessly integrated into various embodiments, providing a more flexible data collection approach. However, these portable devices primarily focus on capturing vision-only demonstration data, limiting their usage for contact-rich and dexterous manipulation tasks where tactile feedback plays a crucial role. In this work, we aim to address both the challenge of efficient data collection and the need for learning more dexterous tasks using visuo-tactile demonstrations. To this end, we introduce ViTaMIn, a novel and effective visuo-tactile manipulation interface designed to capture high-quality demonstrations with enhanced efficiency and flexibility. Unlike conventional approaches that rely on rigid tactile sensors, ViTaMIn leverages an omnidirectional compliant Fin Ray gripper with customized tactile sensing, which can detect contact from all directions as an expressive tactile signal for robot manipulation. We integrate the tactile-aware Fin Ray gripper [ 15 ] with UMI [ 14 ] , enhancing the collected data with rich multimodal information and improving policy learning performance while maintaining the core advantages of portable devices. Additionally, our system enables operators to perceive force feedback during manipulation, facilitating more intuitive and seamless operation. Pre-trained visual representations have shown improved performance in robotic manipulation [ 16 , 17 , 18 , 19 , 20 ] , benefiting from large-scale visual pre-training. To fully leverage the visuo-tactile datasets collected with ViTaMIn, we adopt a multimodal representation learning strategy to pre-train tactile representations, enhancing the robustness and generalizability of our sensor-based policies. Our pre-training objective integrates masked autoencoding [ 21 ] and contrastive learning for multimodal alignment [ 22 ] , where future image observations are aligned with masked current images and tactile signals. Through extensive experiments on five challenging contact-rich manipulation tasks, our visuo-tactile policy, enhanced by multimodal pre-training, exhibits superior data and training efficiency while demonstrating strong generalization across diverse objects and environmental conditions. In conclusion, our contributions are: • ViTaMIn provides a portable and scalable visuo-tactile data collection system. • ViTaMIn proposes an effective multimodal representation learning strategy, which significantly improves the data efficiency, robustness and generalization capabilities. • ViTaMIn achieves superior performance over vision-only baselines across five manipulation tasks by leveraging visuo-tactile demonstrations. II Related Work II-A Visuo-Tactile Manipulation Tactile sensing is essential for robotic manipulation as it provides signals about physical contact in addition to visual observation. Early works [ 23 , 24 , 25 ] use RGB cameras and force/torque sensors to infer contact status for making decisions. However, the information from force/torque sensors is low-dimensional and insufficient for more dexterous manipulation tasks. More recently, vision-based tactile sensors have gained attention for their ability to capture high-resolution contact information [ 26 , 27 , 28 ] . Despite these advances, the rigid design of these sensors restricts the compliance of the end effector, where alternative approaches like uncalibrated tactile skins [ 29 ] and plug-and-play sensing systems [ 30 ] have improved adaptability and flexibility. In our work, we use a Fin-Ray-shaped compliant and all-directional tactile sensor, which can detect contacts from all directions and also support safe and robust contact-rich manipulation. II-B Data Collection System for Robot Manipulation Recent advancements in learning from demonstrations [ 1 , 2 , 3 , 4 ] have shown promising results in developing general-purpose robots. Therefore, efficiently collecting high-quality demonstrations has become a key research focus. Recently works have focused on efficient real-world data collection systems, such as devices or exoskeletons with joint-mapping [ 5 , 6 , 7 ] , exoskeletons [ 8 ] , or vision-based systems [ 9 , 10 ] . However, these approaches require a physical robot during data collection, which limits efficiency and flexibility. In contrast, portable devices [ 11 , 12 , 13 , 14 , 31 , 32 ] offer several advantages: they are low-cost, flexible, and do not depend on a specific physical robot. Additionally, they can be seamlessly integrated into various embodiments and provide a more user-friendly experience for data collection. We extend the UMI data collection system [ 14 ] by integrating tactile sensing, which enriches the demonstrations with multimodal information, improving policy learning performance while preserving the key benefits of portable devices. II-C Multimodal Pre-training for Robotics Pre-trained visual representations have shown improved performance and generalization in robotic manipulation [ 16 , 17 , 18 , 19 , 20 ] with self-supervised learning techniques [ 21 , 22 ] . This can be extended to multimodal representation learning [ 33 , 34 , 35 ] by integrating visual, tactile, and proprioceptive modalities, allowing robots to perceive object properties beyond visual appearance. Aligning heterogeneous sensory modalities is a key challenge in multimodal learning, as different sensors have varying data structures, sampling rates, and noise characteristics [ 36 ] . Inspired by CLIP [ 22 ] , researchers have developed contrastive learning techniques to align tactile and visual representations for manipulation tasks [ 37 , 38 ] . Our work extends these efforts by introducing masked contrastive pre-training, where the tactile encoder learns to reconstruct future occluded visual information, further enhancing multimodal understanding. III Visuo-Tactile Manipulation Interface III-A System Overview