Unlocking Pretrained LLMs for Motion-Related Multimodal Generation: A Fine-Tuning Approach to Unify Diffusion and Next-Token Prediction

Shinichi Tanaka, Zhao Wang, Yoichi Kato, Jun Ohya · 2025

Read Unlocking Pretrained LLMs for Motion-Related Multimodal Generation: A Fine-Tuning Approach to Unify Diffusion and Next-Token Prediction by Shinichi Tanaka, Zhao Wang, Yoichi Kato, Jun Ohya (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06119.txt.

Introduction

Shinichi Tanaka and co-authors turn one pretrained language model into a generator of both words and human motion. Motion generation usually runs on diffusion models, while text runs on next-token prediction; studios want both in a single system that can describe and animate in one breath. Their framework, MoMug, fine-tunes the language model to do both.

Diffusion-style continuous motion output and ordinary text prediction switch seamlessly inside the one architecture. On text-to-motion benchmarks it beat the strongest recent language-model baseline, cutting the standard quality error by more than a third and lifting accuracy across the board. A single model that writes and choreographs simplifies every pipeline for games, film, and virtual avatars.

Catalog record

CREATORShinichi Tanaka, Zhao Wang, Yoichi Kato, Jun Ohya
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06119

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unlocking Pretrained LLMs for Motion-Related Multimodal Generation: A Fine-Tuning Approach to Unify Diffusion and Next-Token Prediction Shinichi Tanaka, Zhao Wang, Yoichi Kato, Jun Ohya Abstract In this paper, we propose a unified framework that leverages a single pretrained LLM for Mo tion-related Mu ltimodal G eneration, referred to as MoMug . MoMug integrates diffusion-based continuous motion generation with the model’s inherent autoregressive discrete text prediction capabilities by fine-tuning a pretrained LLM. This enables seamless switching between continuous motion output and discrete text token prediction within a single model architecture, effectively combining the strengths of both diffusion- and LLM-based approaches. Experimental results show that, compared to the most recent LLM-based baseline, MoMug improves FID by 38% and mean accuracy across seven metrics by 16.61% on the text-to-motion task. Additionally, it improves mean accuracy across eight metrics by 8.44% on the text-to-motion task. To the best of our knowledge, this is the first approach to integrate diffusion- and LLM-based generation within a single model for motion-related multimodal tasks while maintaining low training costs. This establishes a foundation for future advancements in motion-related generation, paving the way for high-quality yet cost-efficient motion synthesis. 1 Introduction Figure 1 : (a) and (b) illustrate the pros and cons of diffusion-based and LLM-based approaches, which motivate us to leverage a pre-trained LLM to take advantage of both. As shown in (c), our MoMug —the first work to unify both approaches in a single pre-trained LLM—significantly improves performance in both text-to-motion and motion-to-text tasks across multiple metrics. Motion-related multimodal generation tasks, including text-to-motion (motion generation) [ 7 ] , motion-to-text (motion captioning) [ 8 , 26 ] , and motion editing [ 2 ] , are crucial for AI and human-computer interaction. These tasks enable machines to generate, understand, and describe human motions, benefiting applications such as film, video games, AR/VR, robotics, and digital humans by enhancing realism and interactivity [ 45 ] . Recent research in motion-related tasks has followed two main trends. The first is diffusion-based methods [ 11 , 27 ] . Since diffusion models are trained to denoise from a Gaussian noise distribution to the target motion distribution [ 5 ] , they inherently focus on maintaining sequence-wide consistency [ 28 ] . Compared to LLM-based methods [ 16 , 38 , 33 ] , which rely on discrete tokenization, diffusion models use DDPM loss, operating in a continuous space. This key distinction allows diffusion models to generate smoother and more natural motion sequences, as they inherently model continuous motion dynamics rather than discrete transitions between tokens [ 39 , 19 ] . Similar advantages have been observed in other generative fields, such as image [ 11 ] , video [ 12 ] , and audio generation [ 15 ] . However, these models also have significant drawbacks: they are limited to generating only the motion modality [ 3 , 38 ] , as diffusion models are primarily designed for generating continuous data (e.g., images, audio, and motion). This limitation makes it impossible to generalize them to other motion-related tasks, such as motion-to-text. The second trend leverages pretrained LLMs [ 21 , 1 ] to generate motion and text within a single model. These approaches [ 38 , 3 , 33 ] benefit from the strong world knowledge [ 41 ] and generalization ability of LLMs, enabling them to produce both motion sequences and textual descriptions in a shared representation space. However, despite their flexibility, these methods have a major limitation: they tokenize motion into a discrete feature space and train the model using a next-token prediction approach (one-after-one supervised fine-tuning), as is common for text. Due to the autoregressive nature of this approach and its reliance on discrete representations [ 40 ] , these models struggle to maintain the global coherence and continuity required for high-quality motion generation. As a result, they often underperform compared to diffusion-based models [ 19 ] , especially when evaluated using metrics that assess generation quality over distribution, such as Multi Modality Distance (MMDist) and Fréchet Inception Distance (FID), which measure the diversity and realism of generated motion. Inspired by recent Multimodal LLM (M-LLM) methods in vision [ 44 ] and video generation [ 17 ] , as illustrated in Figure 1 , we are compelled to ask the following research question: Can we leverage both diffusion and LLMs in a single unified model that can read and generate text while seamlessly switching to a diffusion-based approach for motion generation, all while maintaining low training costs? To answer this question, in this paper, we propose a unified framework that leverages a single pretrained LLM for Mo tion-related Mu ltimodal G eneration, referred to as MoMug . Unlike existing methods that specialize in either diffusion-based continuous motion generation or discrete text and motion prediction, MoMug seamlessly integrates both capabilities within a single LLM model. The model is trained with exposure to both modalities and their corresponding loss functions. By leveraging the shared representation space of an LLM, MoMug ensures better alignment between motion and text, making it adaptable to various motion-related tasks, including text-to-motion generation, motion captioning, and motion editing. We summarize the key novelties of MoMug as follows: • Unified Motion-Text Generation: MoMug is the first framework to integrate diffusion-based continuous motion generation within a single LLM model for motion-related multimodal generation tasks, ensuring that it leverages the advantages of both diffusion and LLM-based methods mentioned above. • Efficient Fine-Tuning Approach: Unlike general diffusion-integrated M-LLM methods that require training from scratch, MoMug leverages pretrained LLMs and applies LoRA-based fine-tuning, significantly reducing computational costs while maintaining high performance. • Superior Performance and Generalization: Experimental results demonstrate that, MoMug improves FID by 38% and mean accuracy across seven metrics by 16.61% on the text-to-motion task. Additionally, it improves mean accuracy across eight metrics by 8.44% on the text-to-motion task. Ablation studies confirm its adaptability across different LLM sizes and its strong generalization ability across various motion-related tasks. 2 Related Works Human Motion Generation Human motion generation is a long-standing research problem that has been extensively studied using various approaches [ 40 ] . Recent advancements focus on motion generation with methods such as VAE-based models [ 23 ] , graph-based techniques [ 34 ] , BERT-based frameworks [ 9 ] , Flow Matching-based methods [ 14 ] , and state-space models [ 42 ] , making this field highly competitive and dynamic. Diffusion based Methods. Diffusion-based methods have become a leading approach for motion generation. These models gradually refine random noise into realistic motion sequences through an iterative denoising process [ 5 ] . This process ensures smooth and temporally consistent motion, which is crucial for text-to-motion tasks [ 28 ] . Several advancements have improved efficiency and flexibility. MDM [ 29 ] applies diffusion directly to motion data, capturing rich temporal dependencies for text-conditioned generation. MLD [ 39 ] reduces computational cost by working in a learned latent space while maintaining quality. MoLA [ 32 ] enhances motion generation and editing by combining latent diffusion with adversarial learning, supporting tasks like in-betweening and path-following. ReMoDiffuse [ 19 ] improves motion refinement by integrating a retrieval mechanism that pulls similar examples from a motion database. These approaches highlight the

Read the complete text