Generative Motion In-betweening by Diffusion over Continuous Implicit Representations
Shiyu Fan, Paul Henderson, Edmond S. L. Ho · 2026
Read Generative Motion In-betweening by Diffusion over Continuous Implicit Representations by Shiyu Fan, Paul Henderson, Edmond S. L. Ho (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12778.txt.
Introduction
Fan, Henderson and Ho improve motion in-betweening, the task of filling motion between sparse keyframes. Recent methods lose keyframe detail and break motion smoothness. Their pipeline maps implicit neural representations to sparse spatial and time cues in a latent diffusion model. From few, ambiguous keyframes, it samples representation parameters and rebuilds plausible smooth motion. Tests show better quality with few keyframes, keeping accuracy while varying the in-between motion.
Catalog record
| CREATOR | Shiyu Fan, Paul Henderson, Edmond S. L. Ho |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52408 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12778
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generative Motion In-betweening by Diffusion over Continuous Implicit Representations Shiyu Fan, Paul Henderson, Edmond S. L. Ho Abstract Recent advances in generative models have yielded impressive progress on motion in-betweening, allowing for more complex, varied, and realistic motion transitions. However, recent methods still exhibit noticeable limitations in preserving keyframe information and ensuring motion continuity. In this paper, we propose a novel pipeline and sampling optimization strategy for latent diffusion models (LDM) based on motion implicit neural representations (INR). By establishing a mapping between INR and sparse spatial or temporal information within latent diffusion, our model can sample the INR parameters from extremely sparse and ambiguous keyframe data and reconstruct plausible and smooth motions from the manifold. Our experiments demonstrate the superior performance of our model, which significantly improves motion generation quality in scenarios with few keyframes while ensuring both keyframe accuracy and diversity of in-between motions. Fig. 1: Given the same set of initial and final keyframes, our model generates a diverse range of in-between motions. I Introduction As a fundamental task of character animation, motion in-betweening involves generating smooth and natural transitions between keyframes in a motion sequence. One of the key challenges is to satisfy the spatial and temporal constraints specified in the keyframes (i.e. target poses) while minimizing unnatural movements like jittering and foot sliding. With the aid of motion in-betweening algorithms, high-quality motions can be generated with only several sparse keyframes as in the examples shown in Fig. 1 . Most recently, motion in-betweening has evolved from a regression task [ 12 ] to a generative task. The emergence of generative models, notably diffusion models [ 16 ] , has brought new life to the traditional motion in-betweening task. Diffusion-based methods have demonstrated promising capabilities for generating diverse and realistic human motions [ 41 , 45 ] . Some diffusion models incorporate motion editing from text-to-motion tasks for motion in-betweening [ 42 , 20 ] . However, existing diffusion models also present significant challenges, which include slow training and generation speeds and wide disparities between sampling results and keyframes. Another limitation of the prior motion in-betweening algorithms [ 6 , 13 ] is low generation quality when the keyframes are extremely sparse. To tackle this problem, models such as CondMDI [ 6 ] require a text description in addition to the keyframes in order to generate high-quality in-between motions. Without such additional input, the results may contain unnatural movement, such as foot sliding or stopping midway through a motion. Such a dependency limits the practicality of the method in scenarios where text conditioning is not available. Additionally, it exhibits large keyframe errors. In order to generate realistic in-between motion from sparse spatial-temporal information, it is essential to utilize a continuous representation to capture global information. By constructing compact and continuous functions, implicit neural representation (INR) captures the global representation of motion sequences and makes it possible to generate smooth and natural motion transitions. Since the encoded motion representation no longer contains explicit spatial or temporal features, inferring the corresponding INR parameters from sparse keyframes is highly challenging. This inspires us to propose a novel latent diffusion sampler for motion INR. Specifically, our model establishes a correlation between randomly selected keyframes and INR latent by the cross-attention module in the denoising network, allowing for INR manifold sampling conditioned on keyframes. To further improve the motion quality and keyframe error, we introduce the concept of manifold preservation for the sampling optimizations and incorporate additional constraints tailored to motion INR. This optimization module, namely Implicit Manifold Guidance (IMG), corrects the sampling direction during the sampling process by minimizing the geometric error and manifold error. Relying on the capability of the implicit neural representation, our model can successfully sample smooth and realistic motion given only sparse, randomly located keyframes. Furthermore, our approach produces motions with reduced keyframe error and which exhibits greater variability. The contributions of this work include: • We are the first to adapt Motion INR to the generative motion in-betweening task. To achieve this, we propose an INR manifold sampler based on latent diffusion model (LDM) within the implicit neural field, which leverages compact continuous-time motion representations to generate realistic and high-quality motion conditioned on extremely sparse keyframes. • We introduce the idea of manifold preservation and design a novel loss in the posterior sampling optimization algorithm tailored for the motion implicit manifold to preserve motion quality while reducing keyframe error. Our method achieves state-of-the-art performance in motion in-betweening tasks conditioned on extremely sparse keyframes without textual descriptions. Explore codes at: https://github.com/Coondinator/NINB II Related Works II-A Motion In-betweening Prior to the emergence of deep learning, the majority of work [ 34 , 5 ] relied on kinematics interpolation to address this problem. Rigid body physics simulation [ 40 , 9 ] and motion imitation [ 23 ] were used for generating plausible in-between frames. With the rise of neural networks, time series models have been applied to the motion prediction task [ 11 , 8 , 39 , 21 , 12 , 27 ] . Qin et al. [ 31 ] stack multi-stage Transformers to progressively generate in-between motion. AvatarGPT [ 47 ] encodes motions as tokens and uses a powerful large language model to achieve both generation and in-betweening tasks. Meanwhile, the concept of high-dimensional manifolds has been gradually adopted in motion encoding and generation [ 18 ] . DeepPhase [ 38 ] presents the periodic phase of the motion manifold, enabling the encoder to capture the periodicity of motions in time. This manifold can also serve as a prior for motion in-betweening [ 37 , 19 ] . Although those auto-regressive methods can provide details by learning from a large dataset, they lack diversity in the generated results. Therefore, there is a growing desire that generative models can bring more diversity to motion synthesis and in-betweening. Petrovich et al. [ 29 ] apply a Variational autoencoder (VAE) to motion synthesis. Ren et al. [ 33 ] synthesize forward and backward motion sequences by introducing a bi-directional stitching CVAE. Due to their superior generative performance, diffusion models have gradually replaced GANs and VAEs. CondMDI [ 6 ] incorporated the diffusion model to handle different types of in-betweening tasks. However, its weak condition constraint results in significant keyframe errors, and learning from discrete-time sequences often leads to abrupt motion transitions. Similar to text-to-motion models, it also relies on text condition inputs to improve motion quality. II-B Motion Diffusion Model and Motion Editing Given the superior performance of diffusion models in image generation tasks, this leading generative model has gradually been applied to the field of motion generation. MDM [ 41 ] and MotionDiffuse [ 45 ] were the first two trials introducing diffusion models into motion generation, and significantly improved the performance of text-to-motion synthesis. Motion latent diffusion [ 2 ] encodes motions into a latent space and performs the diffusion process in this space. This design greatly shortens the training and inference time. Diffusion models have also been introduced into the generation of two-person [ 35 , 24 ] and further extended to multi-person interactions [ 43 ] . Another challenge is how to control the generated results from diffusion. Results generated solely based on prompts