Generative Zoo

Tomasz Niewiadomski 1, Anastasios Yiannakidis 1, Hanz Cuevas-Velasquez 1, Soubhik Sanyal 1, Silvia Zuffi 2, Peter Kulits 1 · 2024

Read Generative Zoo by Tomasz Niewiadomski 1, Anastasios Yiannakidis 1, Hanz Cuevas-Velasquez 1, Soubhik Sanyal 1, Silvia Zuffi 2, Peter Kulits 1 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08101.txt.

Introduction

Tomasz Niewiadomski and co-authors introduce Generative Zoo, a pipeline that creates synthetic training data for estimating animal pose and shape from images. Training such models needs large labeled sets with exact pose and shape notes. Motion capture rigs are impractical for wild animals and cannot scale across many species.

Older fixes fit 3D parameters to hand labeled 2D images, which often yields implausible poses, or use game engines with artist made assets, which lack realism and take much manual effort. The new pipeline renders images with a conditional image generation model instead. Sampled poses and shapes give perfect ground truth labels by construction. The approach aims for diverse, realistic training data that can extend to new species and settings with less effort.

Catalog record

CREATORTomasz Niewiadomski 1, Anastasios Yiannakidis 1, Hanz Cuevas-Velasquez 1, Soubhik Sanyal 1, Silvia Zuffi 2, Peter Kulits 1
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08101

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generative Zoo Tomasz Niewiadomski 1, Anastasios Yiannakidis 1, Hanz Cuevas-Velasquez 1, Soubhik Sanyal 1, Silvia Zuffi 2, Peter Kulits 1 Abstract The model-based estimation of 3D animal pose and shape from images enables computational modeling of animal behavior. Training models for this purpose requires large amounts of labeled image data with precise pose and shape annotations. However, capturing such data requires the use of multi-view or marker-based motion-capture systems, which are impractical to adapt to wild animals in situ and impossible to scale across a comprehensive set of animal species. Some have attempted to address the challenge of procuring training data by pseudo-labeling individual real-world images through manual 2D annotation, followed by 3D-parameter optimization to those labels. While this approach may produce silhouette-aligned samples, the obtained pose and shape parameters are often implausible due to the ill-posed nature of the monocular fitting problem. Sidestepping real-world ambiguity, others have designed complex synthetic-data-generation pipelines leveraging video-game engines and collections of artist-designed 3D assets. Such engines yield perfect ground-truth annotations but are often lacking in visual realism and require considerable manual effort to adapt to new species or environments. Motivated by these shortcomings, we propose an alternative approach to synthetic-data generation: rendering with a conditional image-generation model. We introduce a pipeline that samples a diverse set of poses and shapes for a variety of mammalian quadrupeds and generates realistic images with corresponding ground-truth pose and shape parameters. To demonstrate the scalability of our approach, we introduce GenZoo , a synthetic dataset containing one million images of distinct subjects. We train a 3D pose and shape regressor on GenZoo , which achieves state-of-the-art performance on a real-world 3D animal pose and shape estimation benchmark, despite being trained solely on synthetic data. We will make public our dataset and data-generation pipeline to support future research at https://genzoo.is.tue.mpg.de . Figure 1 : We propose a pipeline for the scalable generation of realistic 3D animal pose and shape estimation training data. Training solely on data produced using our pipeline, we achieve state-of-the-art performance on a real-world 3D pose and shape estimation benchmark. 1 Introduction The estimation of animal pose from images enables the computational modeling of animal behavior [ 1 ] . In quantifying behavior, pose offers a low-dimensional representation amenable to analysis [ 40 ] . From pose, actions can be segmented [ 35 ] or individual health can be monitored [ 8 ] . Pose alone is a fairly descriptive feature when measured in a laboratory setting, where cameras, lighting, and environmental conditions can be tightly controlled. However, pose becomes less informative in the wild, where environmental conditions can vary greatly. Inspired by trends in the modeling of humans [ 32 , 29 ] , recent work has expanded beyond primitive pose-based representations and toward parametric models that represent not only 3D pose, but also shape [ 65 , 3 , 28 , 7 , 67 ] . These representations, typically derived from 3D scans, are steered by body-joint rotation parameters and a latent shape code. However, extracting the parameters of such representations from images is challenging due to the ill-posed nature of 3D inference from 2D images, the diversity of real-world animals, and the variability of imaging conditions in natural environments. Training an effective regression model typically requires large amounts of annotated data. Datasets curated for human pose and shape estimation, for parametric body models like SMPL [ 32 ] , include extensive sequences captured using marker-based mocap systems [ 23 ] or fitted IMU devices [ 50 ] . While humans can be brought into 4D capture halls and outfitted with dense markers, wild animals are less cooperative. As a result, alternative approaches must be devised to overcome the problem of obtaining data. The most common approach is the manual annotation of anatomical 2D landmarks or silhouettes [ 4 ] , which can be used for indirect model supervision. Other approaches begin with 2D annotations and produce pseudo-labels of pose and shape by optimizing a model such as SMAL [ 65 ] to fit the labeled 2D features [ 56 ] . However, while optimizing SMAL to conform to a silhouette may yield 3D fits that appear plausibly aligned, producing accurate 3D annotations is difficult without sufficiently strong priors, as many physically implausible pose and shape combinations could explain the same silhouette. Recently, the use of video-game engines to produce rendered synthetic data has been explored as an alternative to labeling real-world images [ 6 , 20 , 22 ] . While the approach sacrifices visual realism, it offers greater control over dataset curation. Graphics engines’ explicit representation of 3D scenes enables the production of precise ground-truth annotations and control over dataset statistics and variability. However, traditional rendering-based synthetic datasets require substantial manual effort to design or modify. To produce synthetic data for additional species, or render them in a new environment, requires a new set of 3D assets. While synthetic data can be made to appear realistic, achieving both visual realism and sufficient diversity requires considerable resources. We investigate a potentially simpler alternative to the production of synthetic data: rendering with a conditional image-generation model. We propose a pipeline that, given the name of a species, produces paired images and ground-truth pose-and-shape parameters. Rather than relying on explicit collections of 3D assets and scenes, our pipeline is controlled by language: the inclusion of a new species or environmental setting is accomplished via prompting. Our pipeline facilitates the generation of realistic images with a degree of control comparable to traditional synthetic-data generators, thus combining the advantages of visual realism, scalability, and controllable data production. To demonstrate the scalability of our approach, we introduce GenZoo , a million-image dataset comprised of unique poses and shapes across diverse mammalian quadrupeds; see Fig. 1 for examples. Training a regression model solely on our synthetic dataset, without the use of ground-truth real-world images, we achieve state-of-the-art performance on Animal3D [ 56 ] , a real-world animal pose and shape estimation benchmark, validating the quality of our dataset. We also introduce a new synthetic evaluation dataset with greater annotation fidelity than previous benchmarks. In summary, our key contributions include: 1. A scalable pipeline for generating synthetic 3D animal pose and shape estimation data 2. GenZoo , a million-scale dataset of samples produced using our pipeline 3. A state-of-the-art generic animal pose and shape regression model 4. GenZoo-Felidae , a high-quality synthetic test dataset. Figure 2 : Pipeline Overview. Starting with a sampled animal name ( Sec. 3.2 ), we sample corresponding shape parameters ( Sec. 3.3 ). Paired pose parameters are sampled from a set of pseudo-poses ( Sec. 3.4 ). Sampled camera and scene descriptions are combined with a pose caption to form a prompt ( Sec. 3.5 ). Rendered control signals and the prompt are used to guide the conditional image-generation model, resulting in the final image ( Sec. 3.6 ). 2 Related Work Animal Pose and Shape Estimation. The 3D reconstruction of animals follows two primary paradigms: model-free and model-based. Model-free approaches make no assumptions about the animal’s 3D body structure, and the objective is to obtain a representative 3D surface. Given the diversity observed across animal species and shape, this approach is fairly common.

Read the complete text