Achieving Effective Virtual Reality Interactions via Acoustic Gesture Recognition based on Large Language Models

Xijie Zhang, Fengliang He, Hong-Ning Dai · 2025

Read Achieving Effective Virtual Reality Interactions via Acoustic Gesture Recognition based on Large Language Models by Xijie Zhang, Fengliang He, Hong-Ning Dai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07085.txt.

Description

Xijie Zhang, Fengliang He, and Hong-Ning Dai recognize hand gestures for virtual reality from sound instead of cameras. Vision-based recognition is computationally heavy, sensitive to lighting, and raises privacy concerns. Their system emits inaudible high-frequency signals and reads how gestures perturb the returning echoes, captured as channel impulse responses, a low-cost and unobtrusive signal. Earlier acoustic methods needed large labeled training sets, which few-shot VR settings cannot supply.

Theirs is the first system to put a large language model on this task, using differences between impulse responses rather than raw signals to expose the subtle gesture features. They collected a dataset of ten participants performing fifteen gestures, digits, letters, and shapes, ten times each. The language-model classifier matched classic machine learning baselines without domain-certain retraining.

Catalog record

CREATORXijie Zhang, Fengliang He, Hong-Ning Dai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07085

Browse the full library · About the Tower of Babel