DuoGesture: Neuro-Inspired and Biomechanically Informed Dual-Stream Co-Speech Gesture Generation
Ferdinand Paar, Lanmiao Liu, Aslı Özyürek, Serge Thill, Esam Ghaleb · 2026
Read DuoGesture: Neuro-Inspired and Biomechanically Informed Dual-Stream Co-Speech Gesture Generation by Ferdinand Paar, Lanmiao Liu, Aslı Özyürek, Serge Thill, Esam Ghaleb (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26236.txt.
Introduction
In 2026, Ferdinand Paar and four colleagues split co-speech gesture generation into two streams, one carrying meaning and one carrying rhythm, in a system called DuoGesture. A stochastic gate decides frame by frame when meaningful gestures should override beat motion. Motion-grounded conditioning ties semantic choices to movement patterns instead of bare word vectors. An arm-chain beat prior weighted by body measurements steadies the rhythm without dulling meaning. Tests show DuoGesture beating strong baselines on objective scores and human judgments.
Catalog record
| CREATOR | Ferdinand Paar, Lanmiao Liu, Aslı Özyürek, Serge Thill, Esam Ghaleb |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44427 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26236
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DuoGesture: Neuro-Inspired and Biomechanically Informed Dual-Stream Co-Speech Gesture Generation Ferdinand Paar, Lanmiao Liu, Aslı Özyürek, Serge Thill, Esam Ghaleb Abstract Co-speech gesture generation requires both semantic expressivity and biomechanically plausible rhythmic motion. Existing holistic gesture models mix lexically grounded semantic gestures with frequent prosody-aligned beat gestures. This limits semantic grounding, speech-motion alignment, and kinematic smoothness. We propose DuoGesture , a neuro-inspired and biomechanically informed dual-stream approach that decomposes co-speech gesture synthesis into coupled semantic and beat streams. The two streams are coordinated by a Semantic Variational Information Bottleneck , a stochastic frame-level gate that learns when semantic gestures should override rhythmic beat motion. The semantic stream is controlled by Motion-Grounded Semantic Conditioning , which replaces purely linguistic word embeddings with motion-language representations to provide motion-aligned semantic priors for long-tailed lexical triggers of gestures. The beat stream is further regularised by an Inertial Beat Prior , an anthropometry-weighted arm-chain module that reduces jitter and improves rhythmic consistency without constraining semantic frames. Objective evaluations and subjective experiments show that DuoGesture outperforms strong holistic baselines, while component ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation. Demos and qualitative illustrations are available on the project page: https://duogesture.github.io/DuoGesture/ . 1 Introduction Speech and co-speech gestures form an integrated communicative system, yet gesture types play different roles and are processed accordingly. Reliable co-speech gesture generation is therefore important for embodied agents, virtual avatars, and accessible speech-driven animation systems, where gestures affect perceived naturalness, communicative clarity, and user trust. Beat gestures align with prosody and rhythm, supporting turn-taking and the flow of interaction, while semantic gestures (deictic, iconic, metaphoric) are related to lexical content and occur more sparsely [ 23 ] . Cognitive neuroscience further indicates that manual gesture comprehension recruits partially dissociable but interacting networks: parietal–premotor circuits associated with visuomotor structure, and inferior-frontal/temporal circuits implicated in communicative meaning and gesture–speech integration [ 28 , 3 ] . We adopt this dual-process view as a computational stance, where co-speech gesture should be modelled as the interaction of two coupled streams rather than as a single motion process. This stance is different from most current holistic generators, which treat gestures as a single homogeneous stream [ 36 , 18 ] , resulting in three persistent limitations : (i) misalignment between gesture and speech timing, (ii) poor semantic expressivity, and (iii) jittery beat motion that lacks biomechanical principles. We argue that these limitations stem from three modelling drawbacks. First, current architectures rely on a single mechanism to model all gesture frames [ 18 , 16 , 17 , 21 , 20 ] , even though beat and semantic gestures are distinct. Second, semantic conditioning is usually derived from linguistic embeddings, which encode what was said rather than what motion should accompany it . This is especially problematic for long-tailed semantic triggers that appear rarely, or never, during training. Third, beat gestures are learned without physical or biomechanical constraints. These choices create gaps: a semantic gap , where text embeddings provide weak priors for what a gesture should look like, and a beat gap , where rhythmically aligned gestures may achieve reasonable average objective metrics but remain prone to over-smoothing and acceleration artefacts. Figure 1: DuoGesture models co-speech gestures as coupled semantic and beat streams coordinated by a stochastic frame-level weight. It integrates motion-grounded semantic conditioning and an inertial prior to ensure semantic expressivity and rhythmic smoothness. To overcome these limitations and gaps, we propose DuoGesture , a dual-stream model shown in Figure 1 , that separates lexically grounded semantic motion from prosody-aligned beat motion while allowing them to interact. Supported by our auxiliary analysis in Sec. A , which shows distinct temporal and kinematic profiles for semantic- and beat gestures, DuoGesture uses stream-specific conditioning and regularisation rather than a single homogeneous generator. The semantic stream is conditioned by Motion-Grounded Semantic Conditioning (MGSC), which leverages a pretrained text-to-motion representation to provide motion-aligned cues for long-tailed lexical triggers of gestures. The beat stream is regularised by an Inertial Beat Prior (IBP), an anthropometry-weighted arm-chain loss that encourages smooth rhythmic motion while remaining inactive on semantic frames. The streams are coupled by a Semantic Variational Information Bottleneck (S-VIB), which learns when semantic motion should override beat motion. Our novel contributions are summarised as follows: • We formulate a dual co-speech gesture generation model that models lexically grounded semantic motion and prosody-aligned beat motion as coupled but distinct processes. • We introduce MGSC , a motion-grounded semantic conditioning module that uses pretrained text-to-motion representations to improve semantic cues for sparse lexical gesture triggers. • We design IBP , an anthropometry-weighted arm-chain regulariser that reduces beat-motion jitter and improves rhythmic consistency without constraining semantic frames. • We introduce S-VIB , a stochastic frame-level gating mechanism that learns when to activate the semantic stream while avoiding deterministic always-on gate collapse. • We show on BEAT2 that DuoGesture improves distributional realism, as measured by Fréchet Gesture Distance (FGD), in both single- and multi-speaker settings, while maintaining a competitive trade-off across beat alignment, diversity, and facial-motion metrics. 2 Related Work Holistic Co-speech Gesture Generation. Current work has been shaped by two-stage hierarchical quantisation paradigms. Early architectures, e.g., by TalkSHOW [ 32 ] and ProbTalk [ 22 ] , established VAE-based stochastic priors and category-contingent decoding for multifaceted body dynamics. This trajectory toward granularity was extended by EMAGE [ 16 ] , which proposed a spatially-decoupled tokenisation scheme across facial, manual, and corporal streams. SOTA benchmarks are currently defined by PyraMotion [ 33 ] , which leverages a multi-resolution Anchor-based Pyramid VQ-VAE to minimise distributional divergence, yielding a competitive FGD. However, these extant models largely operate under the assumption of kinematic homogeneity , treating gesticulation as a singular stochastic process and thereby blurring the functional divergence between rhythmic prosody and semantic morphology. While SemTalk [ 36 ] introduced a first-order approximation of this divergence via frame-level soft gating, its efficacy remains bottlenecked by shallow linguistic conditioning and a lack of physical regularisation within its homogeneous motion stream. Cross-Modal Semantic Grounding Gesture Generation. A major gap in co-speech gesture generation is the linguistic-kinematic gap , where idiosyncratic word embeddings (e.g., BERT [ 10 ] ) are insufficient to encapsulate the morphology of human motion. To solve this, Semantic Gesticulator [ 37 ] leverages LLM-based lexical parsing, yet the resulting representations remain inherently decoupled from the physical motion manifold. While GestureDiffuCLIP [ 2 ] employs CLIP latents to enforce semantic consistency, it operates within a text-centric latent space, often overlooking the fine-grained temporal dynamics of gesticulation. The RAG-Gesture [ 24 ] introduced non-parametric synthesis by retrieving explicit motion exemplars at inference time. However, lookups in high-dimensional spaces incur significant computational latency and memory usage. Our work resolves this; instead of performing an explicit search, we distil the structured knowledge of a motion-aligned encoder (TM [ 26 ] ) into our generative pipeline. By anchoring semantics directly within a kinematic manifold, DuoGesture achieves the representational richness of retrieval-based methods while maintaining the inference efficiency of purely parametric frameworks, particularly for long-tail lexical distributions. Physically-Consistent Motion Synthesis. While physics-based constraints have been explored in character animation via differentiable simulators or post-hoc projections [ 25 , 35 ] , their integration as inductive biases within gesture models