Scalable Motion In-betweening via Diffusion and Physics-Based Character Adaptation
Jia Qin · 2025
Read Scalable Motion In-betweening via Diffusion and Physics-Based Character Adaptation by Jia Qin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09413.txt.
Introduction
Jia Qin tackles motion in-betweening, the animator's task of filling the frames between sparse key poses. Hand filling is slow, and pure learning methods can drift into poses no real body could hold. The proposed fix pairs a diffusion generator with a physics-based adapter.
In stage one, a diffusion model drafts transitions on a standard skeleton, ignoring who the character is. That draft works across many character shapes at once. In stage two, a reinforcement learning controller retargets the motion to the true character's build and dynamics. It scrubs artifacts and restores style while keeping the motion lawful. Tests on standard sets and stylized figures show smooth, true-to-style results even from sparse, far-apart keys. One model can thus serve a whole cast without retraining.
Catalog record
| CREATOR | Jia Qin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-09413
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scalable Motion In-betweening via Diffusion and Physics-Based Character Adaptation Jia Qin Abstract We propose a two-stage framework for motion in-betweening that combines diffusion-based motion generation with physics-based character adaptation. In Stage 1, a character-agnostic diffusion model synthesizes transitions from sparse keyframes on a canonical skeleton, allowing the same model to generalize across diverse characters. In Stage 2, a reinforcement learning-based controller adapts the canonical motion to the target character’s morphology and dynamics, correcting artifacts and enhancing stylistic realism. This design supports scalable motion generation across characters with diverse skeletons without retraining the entire model. Experiments on standard benchmarks and stylized characters demonstrate that our method produces physically plausible, style-consistent motions under sparse and long-range constraints. August 24, 2026 K eywords Motion In-betweening \cdot Diffusion Models \cdot Physics-based Character Animation 1 Introduction From animated films and video games to virtual avatars and digital twins, high-quality character animation plays a central role in modern digital media. One of the most common and time-consuming tasks in animation production is hand-crafting motion via keyframe animation, with in-between frames generated automatically. Although motion capture technology offers an efficient way to acquire realistic human motion, it remains limited in several important aspects. First, captured data often requires extensive post-processing, such as denoising and cleanup. Second, actors may struggle to perform stylized or physically exaggerated motions that go beyond human capabilities. Third, even when successful, captured motions often need to be retargeted to characters with different skeletal structures or body proportions, introducing further complexity and potential artifacts. These limitations restrict the direct applicability of motion capture in many production scenarios. As a result, manual keyframing and automatic in-betweening remain indispensable tools in modern animation workflows. Recent advances in diffusion-based motion models have significantly improved the quality of in-betweening under sparse keyframe constraints. These models are capable of producing realistic and diverse motions by learning from large-scale human motion datasets. However, most existing methods are built on a fixed skeletal structure [ 1 , 2 ] , making adaptation to new characters challenging and often requiring retraining the entire model. Additionally, many of these approaches operate purely in the kinematic domain [ 3 , 4 ] , lacking physical awareness. As a result, when generating motion for stylized characters, they frequently produce artifacts such as foot sliding, ground penetration, and self-intersections, especially when kinematic retargeting is applied to skeletons with differing proportions. To address these limitations, we propose a two-stage framework that combines the generative flexibility of diffusion models with the physical realism of simulation-based adaptation. In Stage 1, we use a diffusion model to generate motion sequences on a canonical skeleton from sparse keyframes. This stage is character-agnostic, enabling generalization across a wide range of characters. In Stage 2, we train a character-specific controller via reinforcement learning to adapt the canonical motion to the physical structure and dynamics of the target character. This process mitigates physical artifacts and naturally stylizes the motion to reflect the unique morphology of each character. Our main contributions are as follows: • We present a character-agnostic in-betweening framework that removes the dependency on a fixed skeleton and enables generalization across characters with diverse body types and proportions. • We introduce a physics-based motion adaptation stage for motion in-betweening that eliminates artifacts and stylizes the motion to reflect each character’s unique morphology and physical characteristics. We validate our approach on both standard benchmarks and stylized characters. Experiments show that our method produces physically plausible, stylistically consistent, and keyframe-respecting motions across a wide range of constraints. 2 Related Work 2.1 Physically-based Character Animation Physically-based character animation aims to synthesize motions that are both visually realistic and physically plausible by simulating underlying dynamics such as contact, mass, and balance. Compared to kinematic approaches, physics-based methods offer improved realism, particularly in scenarios involving sparse observations or interactive environments. Early methods relied on handcrafted controllers and trajectory optimization [ 5 , 6 ] , but lacked scalability. With the rise of deep reinforcement learning (RL), physics-based control became more generalizable. DeepMimic [ 7 ] demonstrated that RL policies can robustly imitate complex motion clips, inspiring further work on skill learning and motion adaptation. QuestSim [ 8 ] extended this idea to sparse-input tracking, utilizing only headset and controller data to control a full-body avatar in a physics simulator. Reda et al. [ 9 ] further generalized this to real-time motion retargeting for characters with varying morphologies using the same sparse input, enabling cross-skeleton adaptation. MaskedMimic [ 10 ] proposed a unified control framework that learns to inpaint motion segments from diverse control signals (e.g., text, keyframes) in a physics simulation. This allows seamless task-switching under a single model while maintaining physical realism. Beyond RL, Fussell et al. [ 11 ] introduced SuperTrack, a supervised learning approach that replaces PPO with a differentiable physics model to directly optimize tracking policies. This improves training stability and reduces the sample inefficiency typically associated with model-free RL. Recently, physics-based priors have also been integrated into generative models. PhysDiff [ 12 ] introduced a diffusion framework that incorporates physics projection during denoising, producing motions with fewer artifacts such as foot sliding and ground penetration. Unlike post-processing approaches, this integration ensures physical consistency throughout the generative process. Together, these works highlight the importance of physics-based modeling in synthesizing realistic human motion. Despite significant progress, most approaches are restricted to a fixed skeletal structure and few offer generative capabilities for novel motions from sparse keyframes. This motivates the integration of physics-based adaptation with generative motion models, as explored in this work. 2.2 Diffusion Models Diffusion models have emerged as a powerful class of generative models, achieving state-of-the-art performance across a wide range of domains, including image synthesis [ 13 , 14 , 15 , 16 ] , video generation [ 17 , 18 ] , 3D shape modeling [ 19 , 20 ] , and speech synthesis [ 21 ] . Compared to traditional generative models such as GANs and VAEs, diffusion models offer notable advantages, including improved sample diversity, enhanced training stability, and better generalization. These properties make them particularly well-suited for modeling complex, high-dimensional temporal data. A core feature of diffusion models is their iterative generation process, which refines samples from Gaussian noise through a discrete Markov chain or a continuous stochastic differential equation (SDE) formulation [ 22 ] . The Denoising Diffusion Probabilistic Model (DDPM) [ 23 ] established the foundation of this approach, demonstrating that high-quality samples can be generated by reversing a noise-injection process. Subsequent works have introduced various improvements to enhance efficiency and controllability. Classifier-free guidance [ 24 ] improves conditional generation, while latent diffusion models (LDMs) [ 15 ] reduce computational overhead by operating in compressed latent spaces. Other advances, such as flow matching [ 25 ] , score distillation sampling (SDS) [ 26 ] , and denoising diffusion implicit models (DDIMs) [ 27 ] , further enhance generation quality and inference speed. In the field of human motion synthesis, diffusion models have been widely adopted for tasks such as motion generation [ 28 , 29 , 30 , 31 ] and character control [ 32 ] . Notable frameworks like Motion Diffusion Model (MDM) [ 1 ] and MotionDiffuse [ 33 ] leverage diffusion to model