Tempo as the Stable Cue: Hierarchical Mixture of Tempo and Beat Experts for Music to 3D Dance Generation

Guangtao Lyu, Chenghao Xu, Qi Liu, Jiexi Yan, Muli Yang, Fen Fang, Cheng Deng · 2025

Read Tempo as the Stable Cue: Hierarchical Mixture of Tempo and Beat Experts for Music to 3D Dance Generation by Guangtao Lyu, Chenghao Xu, Qi Liu, Jiexi Yan, Muli Yang, Fen Fang, Cheng Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18804.txt.

Introduction

Guangtao Lyu and six co-authors generate 3D dances from music, betting on tempo as the one cue that never lies. Genre labels, used by earlier systems, are noisy or missing and let the dancing drift off the beat. Tempo, by contrast, sits steadily between 60 and 200 beats per minute across datasets. Their model, TempoMoE, posted to arXiv in December 2025, builds rhythm into a diffusion generator.

Motion experts are grouped by tempo range, and multi-scale beat experts track both quick steps and long phrases. A hierarchical routing unit reads the music and blends the right experts on the fly, with no manual labels. Experiments give state-of-the-art dance quality and rhythm alignment. The design lesson travels: anchor generative models to stable physical quantities, not to shaky human annotations.

Catalog record

CREATORGuangtao Lyu, Chenghao Xu, Qi Liu, Jiexi Yan, Muli Yang, Fen Fang, Cheng Deng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18804

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tempo as the Stable Cue: Hierarchical Mixture of Tempo and Beat Experts for Music to 3D Dance Generation Guangtao Lyu, Chenghao Xu, Qi Liu, Jiexi Yan, Muli Yang, Fen Fang, Cheng Deng Abstract Music to 3D dance generation aims to synthesize realistic and rhythmically synchronized human dance from music. While existing methods often rely on additional genre labels to further improve dance generation, such labels are typically noisy, coarse, unavailable, or insufficient to capture the diversity of real-world music, which can result in rhythm misalignment or stylistic drift. In contrast, we observe that tempo, a core property reflecting musical rhythm and pace, remains relatively consistent across datasets and genres, typically ranging from 60 to 200 BPM. Based on this finding, we propose TempoMoE, a hierarchical tempo-aware Mixture-of-Experts module that enhances the diffusion model and its rhythm perception. TempoMoE organizes motion experts into tempo-structured groups for different tempo ranges, with multi-scale beat experts capturing fine- and long-range rhythmic dynamics. A Hierarchical Rhythm-Adaptive Routing dynamically selects and fuses experts from music features, enabling flexible, rhythm-aligned generation without manual genre labels. Extensive experiments demonstrate that TempoMoE achieves state-of-the-art results in dance quality and rhythm alignment. 1 Introduction Music to 3D dance generation aims to automatically synthesize realistic, diverse, and rhythmically aligned human dance conditioned on a given music sequence. This task is fundamental for applications in virtual humans, digital choreography, and audio-visual content creation 47 ; 37 , and also offers insights into the neural coupling of auditory and motor cognition 4 . Recent advances in deep generative modeling, spanning generative adversarial networks (GANs) 23 ; 61 , autoregressive models 48 ; 29 ; 59 , and diffusion models 16 ; 10 ; 34 ; 52 ; 28 , have greatly improved the realism, smoothness, and diversity of synthesized dances. Beyond improving generative architectures, recent works also leverage auxiliary information to enhance dance generation quality, most commonly through genre conditioning. Early approaches represent dance styles using one-hot genre vectors 61 ; 23 ; subsequent works learn continuous genre embeddings to capture richer stylistic variations 48 ; 28 ; more recent studies employ natural language prompts for flexible genre and style control 53 ; 45 . Figure 1 : Visualization of dances under different tempos within the same genre. Even within a single genre, varying BPMs lead to distinct motion patterns: high BPM gives less time per beat, resulting in faster, more localized motions (e.g., quick arm swings, spins), while low BPM allows more time, supporting longer and more complex gestures (e.g., body turns, full-body transitions). Figure 2 : (a) BPM distributions across AIST++, FineDance, and PopDanceSet indicate that musical tempos predominantly fall within a shared range of 60–200 BPM, reflecting a common underlying rhythmic structure. (b–d) In contrast, genre distributions are highly imbalanced and dataset-specific: FineDance and AIST++ adopt distinct genre taxonomies, while PopDanceSet provides no explicit genre annotations. These insights motivate us to leverage BPM as a more reliable cue than genre labels. However, using one-hot genre labels as auxiliary cues for dance generation faces several limitations: (1) Coarse Rhythmic Granularity: dances within the same genre can exhibit widely varying tempo (BPM), intensity, and motion style. High BPM results in faster, more compact movements, while low BPM allows slower, broader, and more intricate gestures. A single genre token is therefore insufficient to capture such intra-genre rhythmic diversity 29 ; see also Figure 1 . (2) Limited Scalability: music and dance continuously evolve, giving rise to new styles, subgenres, and hybrids 2 ; 22 ; 21 . Fixed genre tokens cannot accommodate this open-ended and dynamic style space. (3) Unreliable Supervision: real-world datasets often contain sparse, ambiguous, or inconsistent genre annotations. For example, PopDanceSet 34 lacks genre labels entirely (Figure 2 (d)), making genre-conditioned generation unreliable. These issues limit a model’s ability to accurately capture rhythm and produce high-quality, diverse dance motions. These limitations motivate us to explore alternative cues that can reliably enhance rhythmic dance generation. We find that tempo, a core property reflecting musical pace and typically measured in BPM, remains stable across datasets and genres 39 ; 20 ; 32 . As shown in Figure 2 , despite the diversity of dance styles, BPM cluster within a consistent range (60–200 BPM), revealing a universal rhythmic structure that can serve as a stable prior to enhance the generation model’s perception of rhythm and beat. Motivated by these observations, we exploit the stable tempo cue to improve the diffusion model’s architecture and strengthen its rhythm-aware motion modeling. However, generating tempo-aware motions remains challenging due to the broad and continuous distribution of tempos (Figure 2 ). High-tempo requires rapid and intricate movements, demanding precise modeling of short-term, fine-grained dynamics, whereas low-tempo typically features smooth, continuous gestures evolving over longer durations (Figure 1 ), necessitating modeling of global motion structure 9 ; 5 . Therefore, the generation model struggles to capture diverse tempo characteristics simultaneously, often leading to degraded motion quality or rhythm inconsistencies. To effectively decouple these rhythmic variations, we seek an effective architecture that can adaptively allocate modeling capacity across different tempo conditions. To this end, Mixture-of-Experts (MoE) 17 ; 19 offers a natural solution. MoE decomposes the model into multiple specialized sub-networks called experts, each capable of handling specific input patterns. A learned routing network dynamically selects and combines experts conditioned on input features, allowing the model to scale capacity efficiently and specialize experts across conditions. This clear separation between routing and expert processing has proven effective in vision, speech, and multimodal tasks 31 ; 30 . Building on these insights, we introduce TempoMoE , a tempo-aware MOE framework that incorporates tempo and beat priors into the diffusion model, allowing experts to specialize in distinct BPM ranges and generate motions that are rhythmically aligned and structurally coherent. Concretely, TempoMoE comprises two key components. The first is the Tempo-Structured Expert Groups. We discretize the tempo range into eight 20 BPM bands, as 20 BPM aligns with human perceptual thresholds and common dance practices reported in prior studies 14 ; 13 . Each tempo band corresponds to a dedicated expert group, whose temporal receptive field is designed to align with the motion pacing at that tempo level. Within each group, we define three specialized experts, each operating at a distinct beat scale: quarter-beat, half-beat, and full-beat, capturing rhythmic structures from fine-grained accents to phrase-level transitions. The second key component is a two-stage routing mechanism, termed Hierarchical Rhythm-Adaptive Routing. In the first stage, we perform Hard Tempo-Level Group Selection using a lightweight gating network, TempoGateNet, which predicts the most relevant tempo groups based on the global music representation. We activate the top-2 groups to allow for flexible blending across tempo boundaries. In the second stage, we apply Soft Beat-Scale Expert Routing using BeatGateNet, which assigns soft gating weights to the three intra-group experts, enabling adaptive fusion across beat scales. This hierarchical routing strategy allows the model to dynamically specialize along both tempo and rhythmic resolution dimensions, without relying on genre-specific annotations or fixed genre labels, leading to more accurate and expressive rhythm-aligned dance generation. In summary, our main contributions are as follows: • We identify tempo as a stable auxiliary cue for improving dance generation, remaining consistent across datasets and genres without requiring manual

Read the complete text