Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cues

Varsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera Demberg · 2025

Read Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cues by Varsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera Demberg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03474.txt.

Description

Varsha Suresh and co-authors test, in twenty twenty-five, whether hand gestures help language models follow spoken talks. Speech studies show gestures carry meaning. A hand move can mark a shift in topic, for one. The team asks if modeling gestures and words jointly improves how a model tracks discourse, the flow of a talk. They first turn three-dimensional body motion into discrete gesture tokens, using a learned compression method. Those tokens are then mapped into the same space as word tokens.

They test the joint model on fill-in tasks for three discourse cues: linking words, stance markers, and quantity words. Adding gestures raised accuracy on all three. The authors frame this as a first step toward models that use non-verbal cues.

Catalog record

CREATORVarsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera Demberg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03474

Browse the full library · About the Tower of Babel