MG-Former: A Transformer-Based Framework for Music-Driven 3D Conducting Gesture Generation

Ke Qiu, Yawen Qin, Tianzhi Jia, Xiaole Yang, Kaimin Wang, Kaixing Yang · 2026

Read MG-Former: A Transformer-Based Framework for Music-Driven 3D Conducting Gesture Generation by Ke Qiu, Yawen Qin, Tianzhi Jia, Xiaole Yang, Kaimin Wang, Kaixing Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01197.txt.

Description

Ke Qiu and five co-authors present a transformer framework for generating 3D conducting gestures from music. Conducting synthesis is a hard cross-modal problem: output must follow long-range musical structure, stay synchronized at beat level, and remain plausible as a fine-grained human performance. Earlier studies were limited by sparse pose data, small data, or evaluations that did not directly measure music-gesture alignment.

The authors build a data pipeline that recovers detailed SMPL body motion from conducting videos, forming a dataset of professional gestures. Their system encodes music over time and decodes gestures autoregressively from an initial pose. A retrieval-based evaluation embeds music and motion in a shared space for alignment metrics. Experiments beat dance and conducting baselines, with ablations backing the design.

Catalog record

CREATORKe Qiu, Yawen Qin, Tianzhi Jia, Xiaole Yang, Kaimin Wang, Kaixing Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01197

Browse the full library · About the Tower of Babel