Figure 2
Kaiwen Wang, Kaili Zheng, Yiming Shi, Chenyi Guo · 2025
Read Figure 2 by Kaiwen Wang, Kaili Zheng, Yiming Shi, Chenyi Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13282.txt.
Introduction
Kaiwen Wang and 4 co-authors address the recovery of 3D body meshes for many people from a single photo. Training data for wild scenes is scarce, and older labeling methods treated each person alone. That habit produced crowds with clashing depths and scales inside one image.
Their Depth-conditioned Translation Optimization step refines all crowd placements at once, using height priors and depth cues in a principled framework. Applied to a large dataset, it yields over half a million scene-consistent images. A second network then predicts mesh and camera values in true metric scale, reaching top results on depth reasoning and mesh recovery tests.
Catalog record
| CREATOR | Kaiwen Wang, Kaili Zheng, Yiming Shi, Chenyi Guo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13282
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Figure 2 Kaiwen Wang, Kaili Zheng, Yiming Shi, Chenyi Guo Abstract Multi-person human mesh recovery from a single image is a challenging task, hindered by the scarcity of in-the-wild training data. Prevailing in-the-wild human mesh pseudo-ground-truth (pGT) generation pipelines are single-person-centric, where each human is processed individually without joint optimization. This oversight leads to a lack of scene-level consistency, producing individuals with conflicting depths and scales within the same image. To address this, we introduce D epth-conditioned T ranslation O ptimization (DTO), a novel optimization-based method that jointly refines the camera-space translations of all individuals in a crowd. By leveraging anthropometric priors on human height and depth cues from a monocular depth estimator, DTO solves for a scene-consistent placement of all subjects within a principled Maximum a posteriori (MAP) framework. Applying DTO to the 4D-Humans dataset, we construct DTO-Humans , a new large-scale pGT dataset of 0.56M high-quality, scene-consistent multi-person images, featuring dense crowds with an average of 4.8 persons per image. Furthermore, we propose Metric-Aware HMR , an end-to-end network that directly estimates human mesh and camera parameters in metric scale. This is enabled by a camera branch and a relative metric loss that enforces plausible relative scales. Extensive experiments demonstrate that our method achieves state-of-the-art performance on relative depth reasoning and human mesh recovery. Code is available at: https://github.com/gouba2333/MA-HMR. Towards Metric-Aware Multi-Person Mesh Recovery by Jointly Optimizing Human Crowd in Camera Space Figure 1 : CamSMPLify 41 fits each person independently, causing height and spatial inconsistencies. Our DTO jointly optimizes human translations from height priors and depth cues, ensuring a coherent scene reconstruction. 1 Introduction Recovering 3D human pose and shape from a single image is a fundamental challenge in computer vision with profound implications for applications such as autonomous driving 51 , augmented and virtual reality 56 , sports analysis 33 . While much progress has focused on individuals, real-world scenes are typically composed of multiple people. Challenges like heavy occlusions and depth ambiguities prevalent in multi-person scenes make holistic scene understanding a hard problem to solve. While the construction of in-the-wild 3D human datasets is crucial for training deep learning models, acquiring such data at a large scale remains exceptionally difficult. To overcome this, the community has widely adopted a paradigm of pseudo-ground-truth (pGT) generation 24 ; 20 ; 38 ; 41 . However, current pipelines are single-person-centric. They optimize each individual in isolation and simply assemble the results. This lack of joint optimization ignores scene-level spatial relationships, yielding physically implausible layouts with incorrect depth ordering and inconsistent relative scales (see Fig. 1 , mid). These inconsistent pGTs limit the potential of powerful end-to-end regression frameworks 44 ; 46 ; 2 ; 45 which are capable of learning complex human-human and human-environment relations, and thus place a data bottleneck in the field of multi-person mesh recovery. To resolve scene-wide inconsistencies and put people in their place, we model the problem by mirroring human-like inference, which relies on two key information sources: prior knowledge of human height and visual cues of relative depth. We leverage off-the-shelf models 26 to predict age and gender, from which we derive a statistically-informed height distribution for each individual 17 ; 6 . For the visual cues, we utilize a monocular depth estimator 59 to extract rich geometric information. We then propose a unified framework to integrate these information, named D epth-conditioned T ranslation O ptimization (DTO). DTO solves a joint Maximum a posteriori (MAP) problem, where the height distribution acts as the prior. The likelihood term is conditioned on two robust depth constraints: inter-human depth relations for positioning and intra-human depth scale for structural consistency. The resulting optimization finds the globally optimal human translations that make the entire crowd spatially consistent in the scene (see Fig. 1 , right). We then address the data bottleneck by applying DTO to 2M images from the 4D-Humans dataset 21 ; 54 ; 32 ; 1 and leverage the posterior probability derived from our DTO framework to filter out poorly optimized or inconsistent scenes. This yields DTO-Humans , a large-scale pGT dataset comprising 0.56M high-quality, scene-consistent multi-person images. With 2.7M person instances in dense scenes (avg. 4.8 persons/image), this dataset provides consistent, physically-grounded supervision that has long been a missing piece in training robust multi-person models. Building upon this dataset, we further propose Metric-Aware HMR, an end-to-end network that directly regresses human mesh parameters in metric scale. A critical challenge in metric recovery is the unknown camera intrinsic parameters, which are intricately coupled with the perceived scale of individuals and the overall scene composition. An end-to-end architecture is uniquely suited to resolve this ambiguity, as it can process global image information. Our network materializes this insight by incorporating a camera branch with the HMR model and simultaneously predicts the camera’s field of view (FoV) and human mesh parameters. To guide this joint prediction, we introduce a novel relative metric loss, which explicitly penalizes implausible real-world size relationships among individuals. By learning to predict all parameters concurrently from the full image context, our model learns the complex interplay between scene geometry and individual human scales. In summary, our contributions are as follows: • We propose DTO, a novel MAP optimization framework that jointly optimizes multi-person camera-space translations using both inter-human and intra-human depth constraints to ensure scene-level consistency. • We construct DTO-Humans, a large-scale pGT dataset with high scene consistency, paving the way for training more robust multi-person models. • We design an end-to-end network, Metric-Aware HMR, featuring a dedicated camera branch and a relative metric loss to achieve true metric-scale mesh recovery. • Extensive experiments show that our method achieves state-of-the-art performance on multi-person 3D reconstruction benchmarks, including RH 49 , 3DPW 50 , CMU-Panoptic 19 , Hi4D 60 , and MuPoTS 37 . 2 Related work 2.1 Multi-person Human Mesh Recovery Multi-person human mesh recovery aims to regress all human meshes from a single RGB image. Existing methods can be categorized into multi-stage and one-stage approaches. Multi-stage approaches 30 ; 11 ; 4 ; 41 detect all humans in the image and apply a single-person HMR model to each of them separately. In contrast, single-stage paradigms process the entire image at once to jointly recover all individuals, which allow the model to capture human-human and human-environment interactions. Early methods like ROMP 47 and BMP 61 rely on person-center heatmaps. More recently, DETR 5 ; 28 ; 34 -based frameworks 44 ; 46 ; 2 ; 45 have shown strong performance by using human queries to decode mesh parameters from feature maps. While these end-to-end models have the potential for holistic scene understanding, they highly depend on the quantity and quality of training data. 2.2 Multi-person pGT Generation The scarcity of in-the-wild 3D human data has made pseudo-ground-truth (pGT) generation a cornerstone of modern HMR 20 ; 24 ; 38 ; 25 ; 41 . This paradigm, uses optimization-based methods like SMPLify 35 to create 3D labels for large 2D datasets 32 ; 1 , which then supervise regression networks. However, existing pipelines optimize each person in isolation, leading to scene-inconsistent pGT datasets that directly limit model performance. To improve pGT quality in multi-person scenes, various constraints have been explored. Some methods introduce constraints like