PoseDreamer: Scalable and Photorealistic Human Data Generation Pipeline with Diffusion Models
Lorenza Prospero, Orest Kupyn, Ostap Viniavskyi, João F. Henriques, Christian Rupprecht · 2026
Read PoseDreamer: Scalable and Photorealistic Human Data Generation Pipeline with Diffusion Models by Lorenza Prospero, Orest Kupyn, Ostap Viniavskyi, João F. Henriques, Christian Rupprecht (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28763.txt.
Introduction
Prospero and co-authors present PoseDreamer, a pipeline that generates photorealistic human data with diffusion models. Labeled data for 3D human mesh fitting is hard to collect from single images. Real datasets are small, while rendered datasets lack realism and range. PoseDreamer uses guided generation with preference tuning for control. Step-wise hard sample mining and staged filtering keep quality high. It produced over 500,000 synthetic samples. Models trained on the data matched or beat models trained on real data.
Catalog record
| CREATOR | Lorenza Prospero, Orest Kupyn, Ostap Viniavskyi, João F. Henriques, Christian Rupprecht |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39114 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28763
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PoseDreamer: Scalable and Photorealistic Human Data Generation Pipeline with Diffusion Models Lorenza Prospero, Orest Kupyn, Ostap Viniavskyi, João F. Henriques, Christian Rupprecht Abstract Acquiring labeled datasets for 3D human mesh estimation is challenging due to depth ambiguities and the inherent difficulty of annotating 3D geometry from monocular images. Existing datasets are either real, with manually annotated 3D geometry and limited scale, or synthetic, rendered from 3D engines that provide precise labels but suffer from limited photorealism, low diversity, and high production costs. In this work, we explore a third path: generated data. We introduce PoseDreamer, a novel pipeline that leverages diffusion models to generate large-scale synthetic datasets with 3D mesh annotations. Our approach combines controllable image generation with Direct Preference Optimization for control alignment, curriculum-based hard sample mining, and multi-stage quality filtering. Together, these components naturally maintain correspondence between 3D labels and generated images, while prioritizing challenging samples to maximize dataset utility. Using PoseDreamer, we generate more than 500,000 high-quality synthetic samples, achieving a 76% improvement in image-quality metrics compared to rendering-based datasets. Models trained on PoseDreamer achieve performance comparable to or superior to those trained on real-world and traditional synthetic datasets. In addition, combining PoseDreamer with synthetic datasets results in better performance than combining real-world and synthetic datasets, demonstrating the complementary nature of our dataset. We will release the full dataset and generation code. Figure 1 : PoseDreamer Pipeline: Our method builds a high-quality synthetic human dataset using aligned controllable diffusion, hard sample mining, and filtering. It produces photorealistic images with precise spatial control and diverse scenarios for robust training. With fixed SMPL-X [ 42 ] parameters, varying only the text caption yields diverse, controllable changes in background, clothing, and environment. 1 Introduction Human recognition tasks in Computer Vision encompass a range of methods for detecting and describing humans in images. These range from simpler tasks such as bounding box detection, 2D pose estimation, and semantic segmentation, aimed at identifying the position and semantics of humans in images, to more complex challenges such as 3D detection, 3D pose estimation, and human avatar reconstruction, which seek to unravel the 3D geometry of the human body from images. The success of these methods largely depends on the availability of large labeled datasets for training supervised models. For example, the Microsoft COCO dataset [ 27 ] has significantly advanced 2D human pose estimation. However, creating large labeled datasets is costly and time-consuming, limiting their availability for downstream tasks. Labeling 3D data is even more challenging and prone to errors. For instance, labeling 3D body pose from a single view is highly ambiguous due to depth ambiguities and occlusions. Similarly, the DensePose dataset [ 10 ] incorporated a highly sophisticated labeling process to map pixels to a surface-based representation of the human body, yet produced only very sparse annotations despite significant human effort. This highlights the need for alternative solutions to obtain images with reliable 3D annotations, particularly for human-centric applications. Advanced rendering engines offer one promising direction, allowing researchers to generate synthetic human datasets with corresponding 3D meshes [ 41 , 3 , 7 , 60 ] . However, these approaches face significant practical limitations that restrict their widespread adoption. First, creating photorealistic renders requires substantial technical expertise in 3D modeling, lighting, and material design. Second, building diverse scenes necessitates extensive libraries of 3D assets, including clothing, accessories, backgrounds, and environmental elements, which are expensive to acquire or develop. Third, achieving a realistic human appearance requires sophisticated setups for modeling skin textures, hair, facial expressions, and natural deformations—all of which are computationally intensive. Most critically, rendered images often exhibit a “synthetic look” that creates a domain gap when models trained on such data are applied to real-world scenarios, limiting their practical effectiveness. In this work, we explore an emerging direction: generated data. Recent advances in diffusion models have demonstrated remarkable capabilities in generating photorealistic images across diverse domains. This presents an opportunity to address the limitations of manual annotation and rendering-based approaches by leveraging the inherent realism and scene diversity of generative models. However, the naive application of diffusion models to data generation faces a fundamental challenge: ensuring precise correspondence between generated images and their 3D annotations, a requirement that existing generative models do not inherently satisfy. We present PoseDreamer: a novel pipeline for generating synthetic datasets for human mesh recovery that harnesses state-of-the-art diffusion models [ 24 ] while ensuring precise 3D-2D consistency. We demonstrate that diffusion-based generation is a practical alternative to traditional rendering pipelines, offering greater scene diversity and visual realism. PoseDreamer achieves performance comparable to much larger, manually curated synthetic datasets produced at substantially higher cost, and, when combined, outperforms any other dataset combination. Examples of the generated images and annotations are shown in Figure 2 . The contributions of this work are the following: • Precise Controllable Human Generation: We develop a novel approach for generating photorealistic human images with exact 3D pose control by introducing an enhanced mesh-to-RGB encoding scheme and employing Direct Preference Optimization to align the control model for improved 3D-2D consistency. • Curriculum-Based Generation with Model Feedback: We introduce a two-stage pipeline incorporating feedback from downstream mesh recovery models to prioritize challenging samples through hard sample mining, ensuring maximum learning value while avoiding redundant easy cases. • Large-Scale Synthetic Dataset We introduce a synthetic dataset with 500,000 images, each annotated with a detailed 3D body mesh. This dataset provides a rich and diverse representation for training and evaluating human mesh recovery models. • Synthetic Data Analysis: The experiments validate our synthetic data’s quality, accuracy, and real-world generalization. We show that the model trained on our synthetic dataset achieves results comparable to or better than those of models trained on datasets obtained from 3D game engines. We also show that combining PoseDreamer with synthetic datasets further improves performance over mixing real-world and synthetic datasets, demonstrating the complementary nature of our dataset. This shows that affordable generation with generative models can replace expensive synthetic human-data acquisition. 2 Related Work Figure 2 : Examples from the PoseDreamer Dataset: High-quality synthetic human samples generated using our pipeline. The examples highlight photorealistic appearance, precise spatial control, and diverse, challenging scenarios that support robust model training. We divide the literature review into related methods on data generation with diffusion models and an overview of datasets for human mesh recovery. Data Generation with Diffusion Models: Diffusion models [ 24 , 8 ] , have transformed data generation methods by enabling the creation of high-quality, diverse synthetic datasets. Recently, [ 49 , 47 , 53 , 2 , 62 ] improved the performance of classification models by generating synthetic data with latent diffusion models [ 46 ] . However, the methods are limited to image classification. Diffumask [ 59 ] and DatasetDM [ 58 ] utilize large diffusion models to simultaneously generate synthetic images and annotations for semantic segmentation or depth prediction tasks. Conditioning on text embeddings from large language models [ 44 ] provides a mechanism to encode information outside of the original data distribution. Still, the methods do require finetuning on every specific dataset. Instance Augmentation [ 22 ] generates separate objects in an image, providing a framework to augment and anonymize datasets. VGGHeads [ 21 ] introduces a