How to Train Your Dragon: Automatic Diffusion-Based Rigging for Characters with Diverse Topologies

Zeqi Gu 1,3 Difan Liu 2 Timothy Langlois 2 Matthew Fisher 2 Abe Davis 3 1 Cornell Tech, USA 2 Adobe Research, USA 3 Cornell University, USA · 2025

Read How to Train Your Dragon: Automatic Diffusion-Based Rigging for Characters with Diverse Topologies by Zeqi Gu 1,3 Difan Liu 2 Timothy Langlois 2 Matthew Fisher 2 Abe Davis 3 1 Cornell Tech, USA 2 Adobe Research, USA 3 Cornell University, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15586.txt.

Introduction

Zeqi Gu and four co-authors teach a model to rig drawn figures of many body shapes so they can be posed and moved. Past image moving methods lean on human body maps and large sets of marked real video. Their work came out in twenty twenty-five.

Given just a few frames of a new figure with its bone map, the model infers a rig that can pose it in fresh ways. A made up data pipeline builds varied shapes and skins on the fly for training, with a new bone map form. The team also built the first video set of this kind with both human like and non human figures, each frame marked with key points.

Catalog record

CREATORZeqi Gu 1,3 Difan Liu 2 Timothy Langlois 2 Matthew Fisher 2 Abe Davis 3 1 Cornell Tech, USA 2 Adobe Research, USA 3 Cornell University, USA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How to Train Your Dragon: Automatic Diffusion-Based Rigging for Characters with Diverse Topologies Zeqi Gu 1,3 Difan Liu 2 Timothy Langlois 2 Matthew Fisher 2 Abe Davis 3 1 Cornell Tech, USA 2 Adobe Research, USA 3 Cornell University, USA Abstract Recent diffusion-based methods have achieved impressive results on animating images of human subjects. However, most of that success has built on human-specific body pose representations and extensive training with labeled real videos. In this work, we extend the ability of such models to animate images of characters with more diverse skeletal topologies. Given a small number (3–5) of example frames showing the character in different poses with corresponding skeletal information, our model quickly infers a rig for that character that can generate images corresponding to new skeleton poses. We propose a procedural data generation pipeline that efficiently samples training data with diverse topologies on the fly. We use it, along with a novel skeleton representation, to train our model on articulated shapes spanning a large space of textures and topologies. Then during fine-tuning, our model rapidly adapts to unseen target characters and generalizes well to rendering new poses, both for realistic and more stylized cartoon appearances. To better evaluate performance on this novel and challenging task, we create the first 2D video dataset that contains both humanoid and non-humanoid subjects with per-frame keypoint annotations. With extensive experiments, we demonstrate the superior quality of our results. \ConferencePaper ccs Computing methodologies Animation ccs Computing methodologies Image manipulation † † orcid: 0009-0002-5777-4560 † † orcid: 0000-0001-5971-2748 † † orcid: 0000-0002-5043-8698 † † orcid: 0000-0002-8908-3417 † † orcid: 0000-0003-1469-2696 † † teaser: Workflow. We propose AniDiffusion, a diffusion-based automatic rigging method that helps cartoon creators re-pose a character without going through the traditional arduous rigging procedure. Our method first provides an annotation user interface and asks the user to provide several keyframes of the character with desired control points annotated. Then we invite the user to input the coordinates of those keypoints to indicate how they want to re-pose the character per-frame. After a simple fine-tuning, our model is able to synthesize the character in the new motion. Unlike most existing works, our method does not assume the topology nor the texture of the character belonging to a certain category such as humanoids, and thus works on a much wider range of objects, such as a fox (blue and green box), a pinwheel (orange box, top row), and a dragon (orange box, bottom row). Image credits: fox ©Edina Gecse and pinwheel ©Ekaterine Kantaria; the dragon is self-created. 1 Introduction Animation has always been a labor-intensive process. Before the introduction of digital tools, animation was done by drawing each individual frame by hand, which meant an enormous amount of redundant work went into creating long sequences. Modern software helps reduce this redundant work by representing characters with a hierarchy of parts and corresponding transformations, which are collectively called a rig . Each part in the rig represents geometry that can appear in multiple frames, and each transformation represents an interpolatable way that the character can move. These rigs let artists animate a character by specifying the configuration of a corresponding skeletal representation at a sparse set of keyframes. This representation can greatly accelerate the animation workflow. However, the process of creating an animation rig, called rigging , can be complicated and tedious. It involves factoring the character’s geometry into parts and specifying how each part is affected by transformations in the skeletal hierarchy, including the relative rigidity and bending of different parts in response to motion. To further complicate the process, much of a rig’s information typically resides in metadata that the artist must define using specialized interfaces (e.g., layer decompositions and labels). In this work, we show how image diffusion networks can be trained to infer rigging behavior from a small set of example images and their corresponding skeletal structures. Based on this, we present a tool that greatly simplifies the 2D rigging process. To create a 2D rig, the artist only needs to trace the skeletal structure of one or more example frames showing the character. Then, the artist can draw new unseen skeletal poses, and our tool will generate corresponding images that are consistent with the degrees of freedom and style of deformations observed in the provided examples. Ours is not the first work to create video by conditioning the generation of frames on a reference image and target poses. However, most work in this space has been limited to subjects with standard humanoid topologies–typically, the topology of human pose detectors (e.g., OpenPose [ CHS∗19 ] and DensePose [ GNK18 ] ), which can be used to automatically label abundant training data. A distinct goal of our work is to build a solution that generalizes to more arbitrary and abstract characters, including humanoid characters with more diverse shapes, as well as animals and articulated objects with unseen topologies. In order to support such diversity, we need to adapt a new training strategy and pose representation that is less reliant on automatically-labeled video of humans. Our work makes the following contributions: • We adapt a new pose conditioning strategy that generalizes to more diverse character appearance and skeletal topologies. Here we draw inspiration from the use of texture atlases and texture coordinates. • We present a training strategy that learns to generalize by training on procedurally generated synthetic data, and show that this strategy leads to generalization across real and cartoon images with little fine-tuning (approx. 25 minutes on an NVIDIA A100 GPU). • To foster research and better evaluation in this challenging task, we build the first 2D video dataset spanning both humanoids and non-humanoids with per-frame keypoint annotations to evaluate performance in this generalized problem setting. • We demonstrate that our approach is able to effectively infer the parameters of classical non-neural rigging methods such as As-Rigid-As-Possible (ARAP) [ SA07 ] , and Bounded Biharmonic Weights (BBW) [ JBPS11 ] from the provided examples. This lets us understand our approach as a way to infer and use such rigging behavior from more general image inputs, including hand-drawn example frames. 2 Related Works 2.1 Diffusion-Based Video Synthesis Diffusion models [ SME20 , SSDK∗20 ] were first developed to generate images [ RDN∗22 , NDR∗21 ] . As these image-based models demonstrate increasingly impressive power and are more computationally efficient than their video-based counterparts [ HSG∗22 , HCS∗22 , SCS∗22 ] , how to adapt them for video generation has become an active research topic. AnimateDiff [ GYR∗23 ] expands the dimension on which the model operates from two to three by adding trainable weights for temporal attention between frames. After pre-training, this plug-in module can then be used with most image models to render consecutive dynamic frames. Our model is also image-based, and we include AnimateDiff as an optional element, though we find even without it our model is able to generate outputs stable across time. A popular sub-topic of video generation is video editing, and the most common way to control the editing is via text prompts. Tune-A-Video [ WGW∗23 ] allows changing video content while preserving motions by finetuning a text-to-image diffusion model with a single text-video pair. FateZero

Read the complete text