UniMC : Taming Diffusion Transformer for Uni fied Keypoint-Guided M ulti- C lass Image Generation
Qin Guo, Ailing Zeng, Dongxu Yue, Ceyuan Yang, Yang Cao, Hanzhong Guo, Fei Shen, Wei Liu, Xihui Liu, Dan Xu · 2025
Read UniMC : Taming Diffusion Transformer for Uni fied Keypoint-Guided M ulti- C lass Image Generation by Qin Guo, Ailing Zeng, Dongxu Yue, Ceyuan Yang, Yang Cao, Hanzhong Guo, Fei Shen, Wei Liu, Xihui Liu, Dan Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02713.txt.
Introduction
Telling an image generator where each body joint goes works well for people. It breaks down for animals, and for crowded scenes with overlapping figures. Qin Guo and co-authors build UniMC, a diffusion transformer that packs each instance's class, box, and joint points into compact tokens. The model can then tell figures apart instead of reading a skeleton sketch.
They also release HAIG-2.9M, a dataset of 786,000 images with 2.9 million annotated humans and animals, checked by hand. In tests with heavy overlap and mixed species, UniMC followed the joint guidance more faithfully than earlier methods. One unified controller, it seems, can serve many kinds of subjects.
Catalog record
| CREATOR | Qin Guo, Ailing Zeng, Dongxu Yue, Ceyuan Yang, Yang Cao, Hanzhong Guo, Fei Shen, Wei Liu, Xihui Liu, Dan Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-02713
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UniMC : Taming Diffusion Transformer for Uni fied Keypoint-Guided M ulti- C lass Image Generation Qin Guo, Ailing Zeng, Dongxu Yue, Ceyuan Yang, Yang Cao, Hanzhong Guo, Fei Shen, Wei Liu, Xihui Liu, Dan Xu Abstract Although significant advancements have been achieved in the progress of keypoint-guided Text-to-Image diffusion models, existing mainstream keypoint-guided models encounter challenges in controlling the generation of more general non-rigid objects beyond humans ( e.g. , animals). Moreover, it is difficult to generate multiple overlapping humans and animals based on keypoint controls solely. These challenges arise from two main aspects: the inherent limitations of existing controllable methods and the lack of suitable datasets. First, we design a DiT-based framework, named UniMC , to explore unifying controllable multi-class image generation. UniMC integrates instance- and keypoint-level conditions into compact tokens, incorporating attributes such as class, bounding box, and keypoint coordinates. This approach overcomes the limitations of previous methods that struggled to distinguish instances and classes due to their reliance on skeleton images as conditions. Second, we propose HAIG-2.9M , a large-scale, high-quality, and diverse dataset designed for keypoint-guided human and animal image generation. HAIG-2.9M includes 786K images with 2.9M instances. This dataset features extensive annotations such as keypoints, bounding boxes, and fine-grained captions for both humans and animals, along with rigorous manual inspection to ensure annotation accuracy. Extensive experiments demonstrate the high quality of HAIG-2.9M and the effectiveness of UniMC , particularly in heavy occlusions and multi-class scenarios. Project page can be found at this link . Figure 1 : Top: We establish HAIG-2.9M , a large-scale, high-quality, and highly diverse dataset with joint keypoint-level, instance-level, and densely semantic annotations for both humans and animals. Bottom: Based on HAIG-2.9M , we design UniMC , a controllable DiT-based framework for keypoint-guided image generation, especially for multi-class (e.g., (a), (b), (c)) and heavy occlusion scenarios (e.g., (a), (b)). The bottom part of the figure showcases samples generated by UniMC . 1 Introduction In recent years, diffusion models ( Sohl-Dickstein et al. 2015 ; Ho et al. 2020 ) have gained significant attention due to their remarkable performance in image generation. The generation of non-rigid objects, such as humans and animals, is vital across various domains, including animation ( Zhang et al. 2024 ; Hu et al. 2023 ) , artistic creation ( Midjourney 2024 ) , and animal-related perception tasks ( Yu et al. 2021 ; Yang et al. 2022 ; Shooter et al. 2021 ) . While class-conditioned ( Dhariwal & Nichol 2021 ) and text-conditioned ( Rombach et al. 2022 ; Ramesh et al. 2022 ) image generation has achieved great success, more controllable generation is desirable, especially controlling the structure of non-rigid objects like humans and animals. Previous works on keypoint-guided human image generation ( Zhang et al. 2023 ; Ju et al. 2023b ; Liu et al. 2024b ) have utilized skeleton images as structural conditions. However, it is difficult for those methods to generate multi-class, multi-instance images of humans and/or animals, especially in scenarios with overlapping instances. We identify two primary reasons for this challenge: 1) Conditional formulation limitations : Most methods plot keypoints on images as control signal ( Zhang et al. 2023 ; Ju et al. 2023b ; Liu et al. 2024b ) , as shown in Figure 2 . 2) Dataset limitations : Most existing keypoint-annotated datasets focus on humans ( Lin et al. 2014 ; Ju et al. 2023a ) , with only a few annotating animals ( Yu et al. 2021 ; Yang et al. 2022 ) . None provide keypoint annotations for images featuring both humans and animals. These datasets are designed for perception tasks, lacking the quantity and quality needed for current generative models. Such skeleton image conditions face two main issues: 1) Class binding confusion . The keypoints plotted on the image lack category information. For example, we cannot distinguish if the keypoints should depict a cat or a dog. 2) Instance binding confusion. For images with overlapping instances, it is challenging to distinguish which instance a keypoint belongs to in the overlapping regions. Moreover, variations in skeleton rendering ( e.g. , color, line width) further hinder the generation quality. Consequently, using skeleton images as conditions leads to suboptimal performance for multi-class, multi-instance generation. To overcome the condition formulation limitations, we propose UniMC , a unified Diffusion Transformer (DiT)-based ( Peebles & Xie 2023 ) framework designed for keypoint-guided generation of humans and animals. Specifically, we use each instance’s class name, bounding box and keypoint coordinates instead of skeleton images as the condition signal, thus addressing class binding confusion and instance binding confusion. We then propose unified keypoint encoder , a lightweight keypoint encoder that encodes keypoints of different species into a universal token representation space, avoiding the need for training separate encoders for each species and enhancing both training and inference efficiency. Next, we introduce timestep-aware keypoint modulator , where keypoint tokens are injected into the DiT-based generator to achieve keypoint-level control for multi-class, multi-instance image generation. To overcome the limitations of the dataset, we propose HAIG-2.9M , which includes 786K images with an average aesthetic score ( christophschuhmann 2023 ) of 5.91, covering 31 species classes, and annotated with 2.9M instance-level bounding boxes, keypoints, and captions, averaging 3.66 instances per image. Following ( Yang et al. 2022 ) , these 31 classes can be further classified into 15 animal families following taxonomic rank to facilitate the evaluation of inter-species and inter-family generalization ability of generative models. Among these, nearly half of the images contain animals, and one-quarter of the images feature both humans and animals. Specifically, to ensure data quality and diversity, we crawl 460K images from four high-quality non-commercial data websites ( Pexels 2024 ; Pixabay 2024 ; stocksnap 2024 ; unsplash 2024 ) and filter 2.07M images from four high-quality datasets ( Schuhmann et al. 2024b ; Schuhmann et al. 2024a ; Sun et al. 2024 ; Kirillov et al. 2023 ) . We annotate bounding boxes, human and animal keypoints, and captions using a selection of state-of-the-art (SOTA) models for each annotation type. First, we label 5% of randomly sampled data, then manually evaluate the annotations, selecting the top-performing model for each type. To summarize, our main contributions are three-fold: • We introduce a unified DiT-based framework, UniMC , designed for multi-class, multi-object image generation using keypoint conditions. This framework includes the unified keypoint encoder that encodes keypoints of different species into the shared representation space and the timestep-aware keypoint modulator for effective keypoint-level control. • We propose a unified human and animal image dataset, HAIG-2.9M , for keypoint-guided image generation. This large-scale and diverse dataset includes 786K images and 2.9M instances with comprehensive annotations such as keypoints, bounding boxes, and captions, covering 31 species classes and 15 animal families, facilitating multi-class, multi-object image generation. • Extensive experiments demonstrate the high quality of HAIG-2.9M and the effectiveness and efficiency of UniMC . Figure 2 : Skeleton image conditions face two main issues: (a) Class binding confusion : difficult to distinguish classes from skeleton images alone; (b) Instance binding confusion : challenging to distinguish keypoints of overlapping instances under occlusions ( e.g. , parts in red box ). 2 Related Work Table 1