VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity
Yuemin Mao, Uksang Yoo, Jean Oh, Jonathan Francis, Jeffrey Ichnowski · 2026
Read VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity by Yuemin Mao, Uksang Yoo, Jean Oh, Jonathan Francis, Jeffrey Ichnowski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27344.txt.
Introduction
Mao and colleagues let robot hands feel contact through vibration, then teach policies in simulation on that same signal. Piezoelectric microphones sense those events with high bandwidth, but raw vibration audio is nearly impossible to simulate faithfully. Their VibeAct framework bridges the gap with a shared physical format: per-finger contact and slip. Real recordings are replayed in a digital clone to label that format on its own.
A tactile estimator learns to read contact and slip from real microphone waveforms. Policies train virtually on the identical format computed from simulated contacts. Five contact-rich tasks, from regrasping to insertion, show consistent wins over a proprioception-and-point-cloud baseline in tests. The continuous slip channel proves most informative where reactive control matters. The learned policies carry over to a real dexterous hand-arm platform with improved success.
Catalog record
| CREATOR | Yuemin Mao, Uksang Yoo, Jean Oh, Jonathan Francis, Jeffrey Ichnowski |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27344
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity Yuemin Mao, Uksang Yoo, Jean Oh, Jonathan Francis, Jeffrey Ichnowski Abstract Dexterous manipulation depends on contact events that are fast, local, and often visually occluded. Piezoelectric microphones offer a compact and high-bandwidth way to sense these interactions, but the resulting vibro-acoustic signals are difficult to simulate faithfully enough for end-to-end sim-to-real policy learning on dexterous robot hands. We propose VibeAct , a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning through a shared physical representation of contact and slip. In the real world, we embed piezoelectric microphones into a dexterous robot hand and collect vibro-acoustic data through teleoperation, then replay the recordings in a calibrated digital clone to automatically label per-finger contact and slip. A tactile estimator learns to predict contact and slip from real microphone waveforms, while manipulation policies are trained in simulation on the same representation computed directly from simulated contacts. This decoupling lets policies exploit rapid tactile feedback without simulating raw audio. Across five contact-rich tasks spanning regrasping, in-hand reorientation, and insertion, VibeAct consistently outperforms a proprioception-and-point-cloud baseline in simulation, with the largest gains on tasks requiring sustained reactive control, where the continuous slip-magnitude channel proves the most informative observation. The learned policies transfer to a physical dexterous hand-arm platform, improving success rates on deployed tasks. Project videos and additional details are at vibeact.github.io . Figure 1: VibeAct connects real vibrotactile sensing to simulation-based policy learning through an explicit intermediate representation of contact and slip. A tactile estimator infers this representation from microphone signals, and the policy learns to act on the same representation in simulation. Keywords: Dexterous Manipulation, Tactile Sensing 1 Introduction Open-world manipulation requires rapid responses to contact and slip events [ 29 ] . When a fingertip first makes contact, or when a grasp transitions from sticking to sliding, the relevant hand-object interaction cues are often brief, subtle, or visually occluded [ 7 ] . We present VibeAct , a framework that estimates these contact and slip interactions using piezoelectric microphones embedded in a robot hand and leverages them as tactile observations for learned dexterous manipulation policies. Piezoelectric microphones offer a promising tactile sensing modality for robot hands. Compared with popular vision-based tactile sensors such as GelSight [ 47 ] and Digit [ 15 ] , they are inexpensive, compact, and high-bandwidth. Additionally, they can be mounted away from the contact surface while still capturing structure-borne vibrations generated by impacts, stick-slip motion, and sliding [ 27 , 45 , 16 , 46 ] . The cost of these advantages is that the recorded waveform depends not only on the underlying contact event but also on finger material, adhesive, mounting location, object texture, amplifier chain, and background motor vibrations [ 5 ] . As a result, simulating vibro-acoustic signals accurately enough for end-to-end policy learning is an open challenge. On the other hand, learning dexterous manipulation directly from real-world vibrotactile observations is equally impractical. Collecting large-scale contact-rich demonstrations is labor-intensive, while online reinforcement learning is prohibitively expensive and potentially unsafe due to unstable exploration during early training. Together, these constraints motivate a representation that can bridge real vibrotactile sensing and simulation-based control. The central insight of this work is that contact and slip provide an interface between sensing and control for sim-to-real policy learning. Contact onset, binary slip, and scalar slip magnitude are low-dimensional and task-relevant quantities that can both be estimated from real microphone signals and computed directly in a contact simulator. Rather than training policies on raw audio, VibeAct learns a tactile estimator that maps microphone waveforms to this physical representation, and trains reinforcement learning policies in simulation using the same representation as an observation channel. The estimator solves the sensing problem, while the simulator solves the control problem. We instantiate VibeAct on a dexterous hand mounted to a robot arm, with piezoelectric microphones embedded in each fingertip. We replay real-world recordings of robot trajectories and object poses in a calibrated digital clone, and use the simulator’s contact solver to generate contact and slip labels for training the tactile estimator. Using the same tactile representation alongside proprioceptive and point-cloud observations, we train PPO policies [ 36 ] entirely in simulation. We then evaluate the learned policies on contact-rich manipulation tasks, including in-hand reorientation and peg insertion. This paper makes three contributions: (i) a sim-to-real framework for vibrotactile dexterity based on a shared contact-and-slip representation that bridges real vibrotactile sensing and simulation-based policy learning; (ii) a digital-clone data labeling pipeline that automatically generates per-finger contact and slip supervision from real-world demonstrations; and (iii) an empirical study showing that this representation serves as an effective tactile observation for reinforcement learning across contact-rich dexterous manipulation tasks. 2 Related Work Tactile sensing for dexterous manipulation. Tactile sensing is critical for dexterous manipulation as many task-relevant events occur at the contact interface and are often difficult to infer from vision alone [ 30 , 20 ] . Prior work has explored a wide range of tactile sensing modalities for robotic manipulation. Vision-based tactile sensors have enabled high-resolution reconstruction of contact geometry [ 47 , 15 ] , shear and slip [ 48 , 8 ] , surface texture [ 18 , 26 ] , and contact-rich manipulation behaviors [ 38 , 1 , 13 , 31 , 19 ] . However, scaling them to multi-fingered robot hands introduces challenges including mechanical bulk, camera bandwidth, illumination constraints, and substantial compute overhead. Other tactile sensors, such as magnetic tactile skins [ 2 , 3 , 11 ] and capacitive or resistive sensors [ 41 , 12 , 22 , 42 ] , provide lower-dimensional contact information that can be easier to integrate into control pipelines, but often require modifying the fingertip structure or contact surface material. VibeAct is complementary to these approaches: it uses piezoelectric microphones for high-bandwidth signals, embeds sensors inside fingertips without altering external contact geometry or surface texture, and exposes only a compact tactile representation to the policy. Robot vibrotactile sensing. Contact interactions generate rich structure-borne vibrations that encode information about contact states and interaction dynamics, motivating the use of vibrotactile sensing in robotics [ 25 , 34 ] . Prior works have leveraged vibrotactile sensing for tasks including material property estimation [ 44 ] , contact estimation [ 16 , 49 ] , object and action recognition [ 10 , 21 ] , granular flow estimation [ 6 ] , tool use [ 50 ] , and slip estimation [ 46 ] . A growing body of work instead feeds audio-visual and vibrotactile signals directly into learned manipulation policies, typically through behavior cloning on paired observations and expert actions [ 28 , 39 , 9 , 23 , 17 ] . By training entirely in the real world, these methods avoid simulating vibro-acoustic signals, but they depend on teleoperated or handheld-gripper demonstrations and have not been extended to dexterous multi-fingered hands with embedded microphones, for which such demonstrations are typically difficult to collect. Reinforcement learning with scalable physics simulation offers a promising alternative that removes the need for demonstrations and naturally supports dexterous control [ 33 , 43 ] . Learning dexterous policies directly from raw vibro-acoustic signals in simulation, however, remains impractical, since the