SkelMo: Universal Skeletal Motion Generation for 3D Rigged Shapes

Ye Tao, Yuxin Yao *, Kendong Liu, Dapeng Wu, Junhui Hou · 2026

Read SkelMo: Universal Skeletal Motion Generation for 3D Rigged Shapes by Ye Tao, Yuxin Yao *, Kendong Liu, Dapeng Wu, Junhui Hou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01518.txt.

Introduction

Tao and co-authors generate motion for rigged 3D shapes with SkelMo. Rigged shapes need motion to become 4D assets at scale. Template methods tie to specific topologies and fail on new forms. Per-case tuning is costly and prone to local optima. SkelMo is a diffusion frame for category-free skeletal animation. It learns from 2D video guidance alone. A large dataset of about 20,000 animations supports the training.

Each clip has full textures, skeletal rigging, and rich sequences. A structural-semantic injection bridges 2D cues and 3D skeletons. Texture and semantic traits enter the joint representations directly. The model maps visual dynamics to joint hierarchies and roles. Animations stay anatomically consistent across unseen categories. Tests beat prior methods by a clear margin.

Catalog record

CREATORYe Tao, Yuxin Yao *, Kendong Liu, Dapeng Wu, Junhui Hou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01518

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SkelMo: Universal Skeletal Motion Generation for 3D Rigged Shapes Ye Tao, Yuxin Yao *, Kendong Liu, Dapeng Wu, Junhui Hou Abstract Motion generation for rigged shapes is vital for scalable 4D asset production. However, template-based methods are limited by specific topologies and fail to generalize across diverse morphologies. Conversely, per-case optimization is computationally expensive, susceptible to local optima, and highly sensitive to viewpoint-induced ambiguities. In this paper, we present SkelMo, a diffusion-based framework designed for category-agnostic skeletal animation generation from 2D video guidance. To overcome the scarcity of high-quality training data, we have curated a large-scale dynamic dataset comprising approximately 20,000 diverse 3D animations, each featuring complete textures, skeletal rigging, and a wide array of comprehensive animation sequences. To bridge the kinematic gap between 2D visual motion cues and heterogeneous 3D skeletal structures, we propose a structural-semantic injection mechanism. Our model integrates texture and semantic attributes directly into skeletal joint representations. This allows it to map perceived visual dynamics to specific joint hierarchies and their functional roles. This enables SkelMo to synthesize high-fidelity animations that maintain anatomical consistency across a vast range of unseen categories, from existing biological species to fantastical beings. Extensive experiments demonstrate that our approach significantly outperforms existing methods, setting a new state-of-the-art benchmark for robust and efficient 4D asset generation. Project Page: https://research.davytao.me/skelmo/ . Figure 1 : SkelMo utilizes topology agnostic diffusion to generate skeletal animation according to the given skeleton and driving video. 1 Introduction Generating high-quality motion for rigged objects is a fundamental challenge in computer graphics and computer vision. In the digital realm, this technology addresses a critical bottleneck in 4D asset creation by reducing the reliance on manual animation. As a result, it has the potential to transform immersive entertainment, including video games, virtual reality (VR), and digital avatars. Beyond virtual environments, robust motion generation also serves as a crucial bridge for embodied AI, enabling physical robots to execute realistic and physically plausible movements in the real world. Traditionally, realistic motion is generated through two primary approaches. The first relies on expensive capture systems, such as calibrated multi-camera setups or IMU devices, which require specific environments and are often restricted to certain categories like human bodies. The second approach involves manual animation by professional artists, which is labor-intensive and difficult to scale for diverse virtual characters. Recently, some frameworks [ 41 , 15 , 32 , 26 , 44 , 6 ] have been proposed to automatically synthesize motion from various input modalities, such as text or video. However, due to limited diversity in training data, these methods are primarily restricted to human bodies, quadrupeds, or simple articulated shapes like drawers. AnyTop [ 9 ] introduces an architecture for universal motion generation. However, in the absence of fine-grained conditional control and due to the limited scale of training data (fewer than 100 objects), its practicality remains inherently constrained. With the rapid advancement of video generative models [ 37 , 28 ] , synthesizing motion from monocular videos has emerged as a promising direction. A straightforward approach, as seen in [ 24 , 33 ] , involves optimizing skeletal poses to align the deformed shape with the video frames. Nevertheless, such methods are often sensitive to rapid movements and rely heavily on external priors like depth estimation or point tracking. Furthermore, the requirement for case-specific optimization leads to significant computational overhead. In this paper, we propose SkelMo, a diffusion-based framework for category-agnostic skeletal animation generation. A major hurdle in training such a generative model is that high-quality 3D models with both skeletal rigging and diverse animations are scarce in existing open-source datasets. In response to this data scarcity, we curate a large-scale dynamic dataset consisting of approximately 20,000 carefully selected 3D animations. Each model is equipped with high-fidelity textures, complete skeletal rigging, and a wide array of skeletal animation sequences. The dataset is meticulously preprocessed with rest-pose normalization and orientation alignment to ensure cross-category consistency. Building on this foundation, our video-conditioned model generates skeletal animation parameters for input 3D models, with the ability to generalize across diverse shape types. To achieve true category-agnostic generalization, a key challenge lies in aligning 2D visual motion cues with the diverse 3D skeletal topologies of heterogeneous objects. To address this, we encode the textural and semantic information of each 3D model into its skeletal joint representations, allowing the model to effectively correlate visual motion patterns from the video with object-specific features and achieve better generalization. In summary, our contributions are as follows. • We curate a high-quality dataset of approximately 20,000 3D animations featuring complete textures, skeletal rigging, and diverse animation sequences, all preprocessed for cross-category consistency. • We propose SkelMo, a diffusion-based framework technically featured with a bidirectional video-skeleton fusion and a texture-semantic injection mechanism to enable precise alignment between visual motion cues and diverse 3D morphologies. • We demonstrate that our approach pushes the performance of skeletal motion synthesis to a new height, setting a new state-of-the-art benchmark for generalization across unseen categories and complex motion sequences. 2 Related Work Motion Capture. Motion capture is traditionally categorized into two primary domains: human-centric and non-human domains. For human subjects, state-of-the-art methods predominantly focus on recovering the pose and shape of parametric body models from monocular or multi-view inputs (e.g., HMR [ 11 ] , Vibe [ 13 ] , TCMR [ 5 ] ). A dominant line of work regresses SMPL [ 18 ] or SMPL-X [ 20 ] parameters from images or video clips, leveraging strong template priors to ensure anatomical plausibility and stable tracking. These approaches span optimization-based fitting (e.g., SMPLify-X [ 20 ] ) and modern feed-forward one-stage regressors with ViT-style backbones for robust whole-body recovery (e.g., OSX [ 16 ] , Aios [ 27 ] ). To obtain physically meaningful trajectories, recent monocular systems incorporate SLAM/visual odometry and motion priors to infer world-grounded global motion (e.g., Pace [ 14 ] , Tram [ 29 ] ), yet they still rely on a fixed human template and are not designed for arbitrary skeletons. Non-human motion capture and reconstruction is often studied through model-free (e.g., CMR [ 12 ] , Lassie [ 38 ] , Magicpony [ 30 ] , and Lasr [ 35 ] ) or model-based (e.g., SMAL [ 45 ] ) animal pipelines. Although effective within their target domains, these methods are typically species- or template-specific, which limits their ability to generalize to the diverse and often non-animal skeletal structures found in arbitrary animatable assets. Concurrently, MoCapAnything [ 10 ] introduces a monocular video-based motion prediction method for category-agnostic skeletons. However, its performance is constrained by the reliance on estimated meshes and limited training data. 4D Content Generation. 4D generation, which focuses on dynamic 3D content synthesis, has recently attracted increasing attention. Current 4D generation methods primarily differentiate themselves through various combinations of 3D representations, temporal modeling techniques, and algorithmic paradigms. Early approaches (e.g., Sv4d [ 34 ] , Sv4d2.0 [ 39 ] , 4Diffusion [ 43 ] and DreamGaussian4D [ 21 ] ) predominantly adopted an optimization-based framework to recover dynamic 3D models by lifting 2D video observations into the 3D space. While these methods are category-agnostic, they are often computationally expensive and struggle to maintain spatio-temporal consistency across long

Read the complete text