Visuo-Acoustic Hand Pose and Contact Estimation

Yuemin Mao, Uksang Yoo, Yunchao Yao, Shahram Najam Syed, Luca Bondi, Jonathan Francis, Jean Oh, Jeffrey Ichnowski · 2025

Read Visuo-Acoustic Hand Pose and Contact Estimation by Yuemin Mao, Uksang Yoo, Yunchao Yao, Shahram Najam Syed, Luca Bondi, Jonathan Francis, Jean Oh, Jeffrey Ichnowski (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00852.txt.

Description

Yuemin Mao and co-authors published this robotics paper in 2025. It presents VibeMesh, a wearable system that estimates hand pose and tells exactly where the hand touches an object. Cameras alone struggle here: fingers hide each other, and contact cues are subtle. The team's answer is to add sound. A bone-conduction speaker and small microphones worn on the hand send structured acoustic signals through it, and contact changes how those signals travel.

A graph-based attention network reads the sound patterns together with a hand mesh built from depth camera data. It predicts contact at each point of the mesh. The team also releases a dataset of synced video, depth, sound, and hand-checked contact labels across many handling tasks. In tests, VibeMesh beat vision-only systems, above all where the hand was hidden or holding still against an object.

Catalog record

CREATORYuemin Mao, Uksang Yoo, Yunchao Yao, Shahram Najam Syed, Luca Bondi, Jonathan Francis, Jean Oh, Jeffrey Ichnowski
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00852

Browse the full library · About the Tower of Babel