TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba

Ziyue Yang, Kaixing Yang, Xulong Tang · 2026

Read TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba by Ziyue Yang, Kaixing Yang, Xulong Tang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27314.txt.

Introduction

Yang and co-authors present TokenDance, a two-stage system that generates dance from music token by token. Small 3D dance datasets trap older models in narrow styles and repetitive moves. The first stage tokenizes both music and motion with finite scalar quantization. Dance splits into upper and lower body streams with motion constraints. Music splits into semantic and acoustic features with separate codebooks. A bidirectional Mamba generator then maps music tokens to dance tokens. The local-global-local design keeps motion coherent and tightly music-aligned.

Catalog record

CREATORZiyue Yang, Kaixing Yang, Xulong Tang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39621
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27314

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba Ziyue Yang, Kaixing Yang, Xulong Tang Abstract Music-to-dance generation has broad applications in virtual reality, dance education, and digital character animation. However, the limited coverage of existing 3D dance datasets confines current models to a narrow subset of music styles and choreographic patterns, resulting in poor generalization to real-world music. Consequently, generated dances often become overly simplistic and repetitive, substantially degrading expressiveness and realism. To tackle this problem, we present TokenDance , a two-stage music-to-dance generation framework that explicitly addresses this limitation through dual-modality tokenization and efficient token-level generation. In the first stage, we discretize both dance and music using Finite Scalar Quantization, where dance motions are factorized into upper- and lower-body components with kinematic-dynamic constraints, and music is decomposed into semantic and acoustic features with dedicated codebooks to capture choreography-specific structures. In the second stage, we introduce a Local-Global-Local token-to-token generator built on a Bidirectional Mamba backbone, enabling coherent motion synthesis, strong music-dance alignment, and efficient non-autoregressive inference. Extensive experiments demonstrate that TokenDance achieves overall state-of-the-art (SOTA) performance in both generation quality and inference speed, highlighting its effectiveness and practical value for real-world music-to-dance applications. 1 Introduction With the rapid expansion of internet big data, AIGC tasks have garnered increasing attention from researchers 46 ; 43 ; 45 ; 44 , particularly in the field of AI for Art 40 . Dance is an important form of human artistic expression, while music often provides its structural foundation 25 ; 21 . As a result, the Music-to-Dance generation task has emerged, holding broad application prospects 28 ; 20 in fields such as virtual reality, dance education, and digital character animation. Existing 3D dance generation methods can be broadly categorized into one-stage and two-stage types. One-stage methods directly regress human motion parameters from audio features 21 ; 17 ; 50 . Representative models include Generative Adversarial Network (GAN)-based (CoheDancers 39 ), and Diffusion-based models (EDGE 31 , FineNet 20 , and Lodge 19 ). However, as these methods operate in a continuous regression space without an explicit learned motion prior, they are more susceptible to accumulated errors and manifold drift. Two-stage methods first construct choreographic units and then learn their probability distributions conditioned on music 28 ; 29 , simplifying the generation task into a token-based classification problem. By leveraging strong dance priors, these methods enhance motion plausibility, including Bailando 28 , Bailando++ 29 , MEGADance 37 . However, mainstream 3D dance datasets remain limited in scale, e.g., FineDance 20 with 8 hours, AIST++ 21 with 5 hours, and PopDanceSet 24 with 3.56 hours. Existing methods either directly regress motion in a continuous space or treat music as a continuous conditioning signal, which weakens explicit modeling of fine-grained rhythmic cues and higher-level musical structure. As a result, they are more prone to repetitive phrases and conservative choreography on structurally complex in-the-wild music, as illustrated in Fig. 1 , leading to reduced expressiveness and realism. Figure 1 : Generated dance from in-the-wild music. While music appears infinitely diverse from a listener’s perspective, from a choreographic standpoint it can be abstracted into a finite set of composable elements 25 ; 27 . Specifically, at the semantic level, dance choreography is typically associated with a limited number of style categories (e.g., Popping, Jazz, Hip-hop), while at the acoustic level, music follows a finite set of rhythmic structures (e.g., 2/4, 3/4, and 4/4 time signatures). This observation suggests that, despite surface-level diversity, choreography-relevant musical information lies on a structured and low-dimensional manifold. Motivated by this property, we propose to capture such core features through music tokenization-analogous to dance tokenization in 28 ; 29 -which discovers finite and reusable patterns from complex audio signals and establishes a stable and generalizable prior for dance generation. Following the above observations, we propose TokenDance , a two-stage music-to-dance generation framework designed to explicitly leverage the structured and composable nature of choreography-relevant musical information. In the Dual-Modality Tokenization stage, both music and dance are discretized using Finite Scalar Quantization (FSQ) 26 . For dance, SMPL 23 parameters are factorized into upper- and lower-body motions, with kinematic and dynamic constraints applied during reconstruction to ensure physical plausibility. For music, Librosa-based 21 audio representations are decomposed into semantic features and acoustic features, which are quantized using dedicated codebooks to preserve their heterogeneous characteristics. In the Token-to-Token Generation stage, we introduce a Local-Global-Local generator built upon a Bidirectional Mamba (BiMamba) 48 backbone. Specifically, Music Local Scanners independently encode semantic and acoustic music tokens, while a Global Scanner performs joint fusion and refinement with genre-aware modeling. Finally, two Dance Local Scanners predict upper- and lower-body motion tokens, respectively. The BiMamba backbone efficiently captures long-range contextual dependencies that are essential for modeling intricate music-dance relationships, leading to improved motion coherence and stronger music-dance alignment. Moreover, the proposed generator naturally supports non-autoregressive inference, substantially improving generation efficiency. Figure 2 : Overview of TokenDance. Our contributions to music-to-dance generation are threefold: (1) We propose TokenDance , an efficient two-stage music-to-dance generation framework that achieves state-of-the-art (SOTA) performance in both generation quality and inference speed. (2) We introduce music tokenization in the dual-modality tokenization stage, which substantially improves model generalization. Extensive experiments further demonstrate its model-agnostic effectiveness. (3) We design a Local-Global-Local token-to-token generator, enabling more coherent motion generation and stronger music-dance alignment. Additionally, we adopt FSQ for tokenization and BiMamba as the backbone from prior work, while our main contribution lies in dual-modality tokenization and the Local-Global-Local token-to-token generation design. 2 Related Work 2.1 One-Stage Music-to-Dance Generation Music-driven 3D dance generation has attracted increasing attention due to the strong coupling between musical structure and human motion. Most existing approaches rely on explicit musical representations extracted using audio analysis tools such as Librosa 21 , Jukebox 31 , and MERT 35 , and aim to predict corresponding human motion representations, including SMPL parameters 23 or body keypoints 28 . Early studies typically adopt encoder-decoder architectures to directly regress entire motion sequences from music features 15 ; 30 ; 14 ; 4 ; 21 . While conceptually simple, such approaches often struggle to capture complex spatial dependencies among human joints. To address this issue, later works exploit the hierarchical structure of the human body by incorporating Graph Convolutional Networks (GCNs) 6 ; 34 , which explicitly model joint-level interactions and improve biomechanical plausibility. From a generative modeling perspective, recent advances in AIGC have significantly influenced music-to-dance research. Generative Adversarial Networks (GANs) have been introduced to enhance motion realism by adversarial supervision, where discriminators guide generators toward more natural and expressive dance motions 3 ; 12 ; 39 . Recently, Diffusion Models have shown remarkable success in various AIGC tasks, with notable applications extending to the music-to-dance domain 31 ; 20 ; 19 ; 18 ; 38 ; 41 , but the computational cost of the sampling process remains high. Overall, one-stage methods directly regress motion from music in a continuous space, making them more prone to accumulated errors and manifold drift during inference, especially for long sequences. This motivates two-stage formulations, whose main advantage is the learned discrete latent space or codebook built from real human motion, which provides a stronger motion prior for generation and thus improves motion stability. 2.2 Two-Stage

Read the complete text