Achieving Effective Virtual Reality Interactions via Acoustic Gesture Recognition based on Large Language Models
Xijie Zhang, Fengliang He, Hong-Ning Dai · 2025
Read Achieving Effective Virtual Reality Interactions via Acoustic Gesture Recognition based on Large Language Models by Xijie Zhang, Fengliang He, Hong-Ning Dai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07085.txt.
Description
Xijie Zhang, Fengliang He, and Hong-Ning Dai recognize hand gestures for virtual reality from sound instead of cameras. Vision-based recognition is computationally heavy, sensitive to lighting, and raises privacy concerns. Their system emits inaudible high-frequency signals and reads how gestures perturb the returning echoes, captured as channel impulse responses, a low-cost and unobtrusive signal. Earlier acoustic methods needed large labeled training sets, which few-shot VR settings cannot supply.
Theirs is the first system to put a large language model on this task, using differences between impulse responses rather than raw signals to expose the subtle gesture features. They collected a dataset of ten participants performing fifteen gestures, digits, letters, and shapes, ten times each. The language-model classifier matched classic machine learning baselines without domain-certain retraining.
Catalog record
| CREATOR | Xijie Zhang, Fengliang He, Hong-Ning Dai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07085