Expotion: Facial Expression and Motion Control for Multimodal Music Generation
2025
Read Expotion: Facial Expression and Motion Control for Multimodal Music Generation (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04955.txt.
Introduction
Fathinah Izzati, Xinyue Li, and Gus Xia build a music generator steered by faces and movement, in a 2025 paper. Their system, Expotion, takes a video of a person's expressions and upper-body motion, plus a text prompt, and composes music meant to match the mood and timing of the performance.
They adapted a large text-to-music model with a light fine-tuning method, so only a small dataset was needed. A smoothing step keeps the video and the music in step over time. In tests, adding the visual signals improved musicality, creativity, and beat matching, and the output followed both video and text better than rival systems, including dedicated video-to-music models. The team also released seven hours of recorded performances with matching music to seed further work.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-04955
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Expotion: Facial Expression and Motion Control for Multimodal Music Generation Abstract We propose Expotion (Facial Exp ression and M otion Control for Multimodal Music Generation), a generative model leveraging multimodal visual controls—specifically, human facial expressions and upper-body motion—as well as text prompts to produce expressive and temporally accurate music. We adopt parameter-efficient fine-tuning (PEFT) on the pretrained text-to-music generation model, enabling fine-grained adaptation to the multimodal controls using a small dataset. To ensure precise synchronization between video and music, we introduce a temporal smoothing strategy to align multiple modalities. Experiments demonstrate that integrating visual features alongside textual descriptions enhances the overall quality of generated music in terms of musicality, creativity, beat-tempo consistency, temporal alignment with the video, and text adherence, surpassing both proposed baselines and existing state-of-the-art video-to-music generation models. Additionally, we introduce a novel dataset consisting of 7 hours of synchronized video recordings capturing expressive facial and upper-body gestures aligned with corresponding music, providing significant potential for future research in multimodal and interactive music generation. Code, demo and dataset are available at https://github.com/xinyueli2896/Expotion.git 1 Introduction Music generation models have become increasingly versatile and interactive, capable of integrating control signals from various modalities, including audio, text, and symbolic representations (such as MIDI or musical scores). These conditions act as control to guide the model toward producing more precise, and targeted outputs aligned with user expectations. Although current text-to-music generation models can produce impressive musical quality, they often lack the fine-grained temporal control mechanisms and expressivity necessary to flexibly adapt to a wide range of real-world scenarios. Inspired by the idea that gestures and facial expressions can act as important guides for music, similar to what conductors do, we further explore visual controls in this study and propose Expotion , a deep music generative model with multimodal controls–facial exp ression and upper body m otion , as well as text prompts. Expotion is designed to synthesize high-quality music that is both expressive (ensuring that the musical content reflects the emotional and expressive cues of the face and gestures) and temporally accurate (so that every change in motion and expression is accurately mirrored in the music) with the input video, as shown in Figure 1 . Figure 1 : Overview of Expotion’s multimodal inference pipeline, showing how visual gestures and facial expressions guide expressive music generation. While some studies have explored audio–music generation tasks given videos—ranging from audio effects generation (e.g., [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] ) video music background generation (e.g., [ 12 , 13 , 14 , 15 ] ), and dance music generation (e.g., [ 16 , 17 , 18 ] among others), our work focuses on the subtle dynamics of gestures and facial expressions, emphasizing their fine-grained temporal synchronization with music generation. This opens up potential applications in real-time, interactive audiovisual systems. To achieve our goal, we first introduce a newly curated dataset comprising of 7 hours of carefully synchronized video-music pairs featuring expressive gestures and facial expressions closely matched to the corresponding music. Given the limited amount of data, we employ parameter-efficient fine-tuning (PEFT) on a transformer-based text-to-music generation model [ 19 ] —leveraging powerful ability of the model pretrained on a massive music-text data that have been shown to be effective in incorporating additional modalities [ 20 , 21 , 9 , 10 , 22 , 23 ] . By fine-tuning only 4% of the original model’s parameters [ 20 ] , our method seamlessly integrates multimodal visual inputs (facial expressions and upper-body gestures) using only 130 video–audio pairs for training, thereby minimizing network complexity while ensuring robust multimodal fusion. We also propose an approach called temporal smoothing to ensure precise and efficient temporal alignment between audio and video modalities. Our experiments show that Expotion can generate high-quality, temporally accurate music that faithfully reflects the expressive nuances inherent in the visual inputs and textual descriptions. Although textual descriptions supply the primary contextual cues—leveraging the model’s strong text-understanding capabilities to guarantee baseline music quality—the addition of visual input further enhances alignment, expressiveness, and consistency. In comprehensive subjective and objective evaluations, Expotion consistently outperforms current state-of-the-art video-to-music generation models [ 24 ] and multimodal captioning baselines across multiple metrics, including (1) Quality of Generated Music, (2) Beats and Tempo Consistency, (3) Text-Audio Similarity, and (4) Video–Music Consistency. To the best of our knowledge, this work is the first to leverage synchronized expressive gestures and facial expressions for music generation. Our experiments show that incorporating visual features as control signals not only enhances the temporal alignment between the video and generated music but also improves text adherence and overall musical quality—highlighting the complementary strengths of both modalities. We believe that Expotion will empower artists with a more expressive, controllable, and interactive approach to music creation. Figure 2 : Joint embedding of visual features. Facial expression and motion features are first extracted from the given video and temporally smoothed through interpolation, producing refined intermediate representations. These temporally aligned embeddings then undergo dimensionality reduction through the low-rank projection layer. Then, the concatenated embeddings are projected to the same dimension as the MusicGen hidden layers. Finally, the positional encoding is added to the latent embeddings to form the final joint embeddings. 2 Related Work We review three key paradigms for controllable music generation: visual and motion-based control, textual and symbolic conditioning, and training and adaptation strategies. Visual and Motion-Based Control Early interactive systems mapped facial or bodily features directly to sound. Valenti et al.’s Sonify Your Face modulated audio via Bayesian classification of facial motion units [ 25 ] , and Clay et al. translated whole-body emotional expressions into electronic-music parameters [ 26 ] . D2MNet extracted global style and local beat vectors from LMA-derived movement signals to drive an autoregressive generator [ 27 ] . DeepTunes [ 28 ] and [ 29 ] combine CNN-based emotion detection with GPT-2 lyric models, LSTMs, and transformers to jointly predict discrete and valence–arousal emotions and produce synchronized music (and lyrics) that closely reflect the user’s image input. Video-conditioned models such as VidMuse [ 24 ] , V2Meow [ 15 ] , and Video2Music [ 14 ] align music with motion cues and scene context, while Foley-style systems translate individual events to sound via latent diffusion or transformers [ 2 , 8 , 1 ] . Textual and Symbolic Conditioning Text-to-music models like MusicLM [ 30 ] and MusicGen [ 19 ] generate high-quality audio from descriptive prompts but lack explicit time-varying control. MusicGen-Melody extends this by conditioning on a reference melody track for pitch contours [ 19 ] . CoCoMulla leverages chord charts and drum patterns to control harmony and rhythm [ 20 ] , and Sketch2Sound introduces continuous vocal–imitation curves (loudness, brightness, pitch) alongside text to modulate audio generation [ 21 ] . Training and Adaptation Strategies Many video-to-audio and music models are trained from scratch on large paired datasets—such as AudioSet [ 31 ] and VGGSound [ 32 ] —to learn cross-modal correspondences [ 1 , 2 , 3 , 4 , 24 ] . When data is limited, parameter-efficient fine-tuning (PEFT) is effective: Sketch2Sound fine-tunes a single linear layer per control signal on