Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures
Varsha Suresh, Mohammad Mahdi Abootorabi, Mohamed Salman M. Hamza Mughal, Christian Theobalt, Ashwin Ram, Jürgen Steimle, Vera Demberg · 2026
Read Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures by Varsha Suresh, Mohammad Mahdi Abootorabi, Mohamed Salman M. Hamza Mughal, Christian Theobalt, Ashwin Ram, Jürgen Steimle, Vera Demberg (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30608.txt.
Introduction
Suresh and colleagues bridge spoken words and co-speech gestures with semantic motion anchors, plain labels capturing a gesture's form and communicative intent. Direct matching of transcripts with flowing motion embeddings stresses low-level movement too much and misses what gestures mean. Their method cuts three-dimensional gestures into body-hand units, writes them up as set notes, and grounds them in the transcript for extra contrast training. On BEAT2, text-to-gesture search lifts clearly over a direct baseline, and users prefer the retrieved gestures for new ones.
Catalog record
| CREATOR | Varsha Suresh, Mohammad Mahdi Abootorabi, Mohamed Salman M. Hamza Mughal, Christian Theobalt, Ashwin Ram, Jürgen Steimle, Vera Demberg |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 103789 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30608
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures Varsha Suresh, Mohammad Mahdi Abootorabi, Mohamed Salman M. Hamza Mughal, Christian Theobalt, Ashwin Ram, Jürgen Steimle, Vera Demberg Abstract Learning a shared representation between spoken text and gesture is central to co-speech gesture retrieval, synthesis, and understanding, but remains challenging for semantically meaningful gestures whose communicative intent is not captured by motion alone. Direct contrastive alignment between transcripts and continuous motion embeddings often overemphasizes low-level kinematics and misses the symbolic content of semantic gestures. We propose semantic motion anchors , natural-language abstractions of gesture motion capturing physical form and communicative intent. Our method discretizes 3D gestures into body-hand motion primitives, verbalizes them into structured descriptions, and grounds them in the transcript to provide auxiliary contrastive supervision. On BEAT2, our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline and outperforms prior retrieval approaches on text to gesture and gesture to text retrieval directions. Beyond aggregate retrieval metrics, semantic motion anchor supervision helps retrieve gestures that are semantically meaningful for the spoken query, rather than defaulting to generic motion patterns. A downstream retrieval-augmented gesture generation study showed that users significantly preferred gestures retrieved by our approach over a retrieval-augmented generation baseline, demonstrating that semantically grounded retrieval translates to gestures that better convey communicative intent in downstream generation. * * footnotetext: These authors contributed equally to this work. † † footnotetext: This author is supported by the Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA) through the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research. 1 Introduction Gestures are a core channel of human communication. Semantically meaningful co-speech gestures can complement or reinforce spoken content, make communication more effective, and are central to tasks such as co-speech gesture synthesis and understanding ( 15 ) . A key requirement underlying these tasks is learning a shared space between language and gestures that meaningfully aligns raw motion sequences with spoken language. Learning such a space that adequately captures semantic gestures, however, remains challenging. Approaches that directly map raw motion sequences with spoken text often fail to capture higher-level semantics, instead learning averaged representations that are dominated by frequent beat gestures ( 15 ; 23 ; 1 ; 9 ; 4 ; 14 ) . This is because semantic gestures are sparse and lie in the long tail of natural human motion distributions ( 15 ) , making them underrepresented despite their importance for conveying communicative intent. Consequently, retrieval-based approaches have been proposed to inject semantically relevant gestures into the generation process ( 22 ; 14 ) . However, these retrieval strategies are typically based on heuristic or rule-based matching, or more recently, on learning raw motion-to-text mappings specifically for semantic gestures ( 4 ) , and thus remain limited in effectively modeling semantic gesture alignment. A central challenge underlying these approaches lies in how gestures and spoken language are represented and mapped. Most existing methods learn motion embeddings under reconstruction-based objectives, which emphasize low-level kinematic features, but these are often not directly aligned with communicative intent, which is highly relevant for semantic gestures. For instance, a semantic gesture conveying enumeration (“first, second, third”) can differ significantly in articulation across speakers, yet express the same meaning. Conversely, semantic gestures with similar motion patterns may encode entirely different intents depending on discourse context. This mismatch highlights a core limitation of current approaches: they conflate similarity in physical motion with similarity in semantic meaning, making it difficult to learn representations that generalize across the sparse and diverse space of semantic gestures. In this work, we argue that semantic gesture retrieval should not rely solely on directly mapping spoken text to continuous motion space, but should be supported by a semantically relevant abstraction that can better link the two modalities. To this end, we introduce semantic motion anchors : structured natural-language descriptions that re-express motion abstracted in terms of physical form and communicative intent. Here, physical form refers to gesture-relevant properties such as handedness, spatial position, motion trajectory, and hand configuration ( 7 ) , rather than raw frame-level joint coordinates. Communicative intent captures the gesture’s contextual function, such as listing, self-reference and uncertainty. Together, these anchors preserve motion aspects that matter for interpretation while reducing sensitivity to low-level kinematic variation that may be irrelevant for learning the shared space. Our approach consists of three main components: First, we train a two-stream RVQ-VAE ( 20 ; 8 ) to compress continuous 3D gesture sequences into discrete motion token, and deterministically map each primitive to a structured natural-language fragment describing observable spatial and kinematic properties such as hand position, movement direction, handedness, and hand configuration taken from 7 . Second, we use an LLM to compose these token-level descriptions with the speech transcript into semantic motion anchors for each gesture. Third, we use these anchors as auxiliary supervision in contrastive text-gesture motion retrieval training. We hypothesize that this allows the model to learn relevant details required for retrieval, i.e., gesture form and function, making the mapping between spoken language and motion more semantically grounded. On BEAT2 ( 8 ) , our method improves text-to-gesture R@1 from 39.1 to 42.3, a +3.2 point absolute gain, corresponding to an 8.2% relative improvement over the direct text-gesture motion baseline. In a downstream retrieval-augmented gesture generation study, users significantly preferred gestures retrieved by our approach over those from RAG-Gesture 14 (72.2% vs. 27.8%, p<0.0001 ), demonstrating that semantically grounded retrieval translates to gestures that better match communicative intent in practice. Our contributions are: (i) We introduce semantic motion anchors for text-gesture retrieval, representing co-speech gestures through natural-language descriptions of form and intent. (ii) We propose an anchor-supervised contrastive learning framework that uses motion-token verbalization and transcript grounding to improve language-gesture alignment. (iii) We release Semantix , a dataset of 878 human-annotated TED and BEAT2 clips with gold form and intent descriptions for evaluating semantic gesture understanding. (iv) We demonstrate gains on BEAT2 retrieval, TED–BEAT2 cross-dataset semantic retrieval, and downstream user preference against RAG-Gesture. 2 Related Work 2.1 Co-Speech Gestures Co-speech gestures are an integral part of spoken communication and convey meaning jointly with speech ( 12 ; 6 ) . Gesture studies commonly distinguish representational gestures, such as iconic, metaphoric, and deictic gestures, from beat gestures, which primarily mark rhythm ( 12 ; 13 ) . This distinction is important for computational modeling: beat gestures are frequent and relatively well captured by speech-synchronized motion models, whereas semantic gestures are sparse, context-dependent, and often lie in the long tail of natural gesture distributions ( 15 ; 19 ; 14 ) . Recent co-speech gesture generation methods notably improve naturalness and temporal alignment by modeling speech-conditioned motion, but often produce generic or beat-dominated gestures ( 15 ; 23 ; 1 ) . To address this, semantics-aware approaches introduce language-motion alignment, semantic planning, or retrieval-augmented generation to produce gestures that better match discourse meaning ( 1 ; 23 ; 22 ; 14 ; 9 ; 19 ) . These methods suggest that semantic grounding is important for communicatively meaningful generation. However, because their primary focus is synthesis, retrieval is often treated as an intermediate step and implemented using rule-based or