Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cues
Varsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera Demberg · 2025
Read Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cues by Varsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera Demberg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03474.txt.
Description
Varsha Suresh and co-authors test, in twenty twenty-five, whether hand gestures help language models follow spoken talks. Speech studies show gestures carry meaning. A hand move can mark a shift in topic, for one. The team asks if modeling gestures and words jointly improves how a model tracks discourse, the flow of a talk. They first turn three-dimensional body motion into discrete gesture tokens, using a learned compression method. Those tokens are then mapped into the same space as word tokens.
They test the joint model on fill-in tasks for three discourse cues: linking words, stance markers, and quantity words. Adding gestures raised accuracy on all three. The authors frame this as a first step toward models that use non-verbal cues.
Catalog record
| CREATOR | Varsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera Demberg |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-03474