Robust Human Registration with Body Part Segmentation on Noisy Point Clouds

Kai Lascheit Daniel Barath Marc Pollefeys Leonidas Guibas Francis Engelmann, ETH Zurich Microsoft Stanford University Google · 2025

Read Robust Human Registration with Body Part Segmentation on Noisy Point Clouds by Kai Lascheit Daniel Barath Marc Pollefeys Leonidas Guibas Francis Engelmann, ETH Zurich Microsoft Stanford University Google (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03602.txt.

Introduction

Kai Lascheit and co-authors fit a digital human body to noisy 3D point clouds with the help of body part labels, in a paper from 2025. Matching a mesh to a scan is key for augmented reality and robots that work near people. Real scans carry noise and clutter, which often throw the fit off.

Their method first labels each point by body part. Those labels guide a two-step fit of the SMPL-X body model, starting from part centers for pose, then refining the whole match. The fitted mesh can then clean up the labels, lifting segmentation scores too. On three messy real data sets, InterCap, EgoBody, and BEHAVE, the method beat prior work on both pose and segmentation.

Catalog record

CREATORKai Lascheit Daniel Barath Marc Pollefeys Leonidas Guibas Francis Engelmann, ETH Zurich Microsoft Stanford University Google
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-03602

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust Human Registration with Body Part Segmentation on Noisy Point Clouds Kai Lascheit Daniel Barath Marc Pollefeys Leonidas Guibas Francis Engelmann, ETH Zurich Microsoft Stanford University Google Abstract Registering human meshes to 3D point clouds is essential for applications such as augmented reality and human-robot interaction but often yields imprecise results due to noise and background clutter in real-world data. We introduce a hybrid approach that incorporates body-part segmentation into the mesh fitting process, enhancing both human pose estimation and segmentation accuracy. Our method first assigns body part labels to individual points, which then guide a two-step SMPL-X fitting: initial pose and orientation estimation using body part centroids, followed by global refinement of the point cloud alignment. Additionally, we demonstrate that the fitted human mesh can refine body part labels, leading to improved segmentation. Evaluations on the cluttered and noisy real-world datasets InterCap, EgoBody, and BEHAVE show that our approach significantly outperforms prior methods in both pose estimation and segmentation accuracy. Code and results are available on our project website: https://segfit.github.io Input Point Cloud Ground Truth ArtEq [ 11 ] NICP [ 21 ] SegFit (Ours) Figure 1 : Our method SegFit reconstructs human poses from point clouds using body part segmentation and the SMPL-X model [ 23 ] . We showcase SMPL-X fitting results on the EgoBody dataset [ 31 ] , and compare to the state-of-the-art methods ArtEq [ 11 ] and NICP [ 21 ] . 1 Introduction Registering parametric human meshes to 3D point clouds requires estimating both human pose and shape parameters. Accurately capturing the nuances of human movement and form is essential for various applications, from generating realistic virtual avatars [ 27 , 25 ] to enhancing human-robot interactions [ 2 , 18 ] . While fitting a parametric human body model to 3D data (e.g., from LiDAR or Kinect) is a crucial step in these applications, the absence of contextual cues and the presence of noisy, cluttered environments often result in suboptimal fits. In contrast, this work explores how leveraging body parts can significantly improve fitting accuracy. Towards that end, we introduce a novel method that integrates body part segmentation with pose fitting for partial, noisy, and cluttered 3D point clouds. In recent years, parametric models like SMPL [ 19 ] and its extension SMPL-X [ 23 ] have become the de facto standard for describing 3D human poses and shapes. These models are typically fitted to point cloud data using either gradient-based optimization [ 10 , 6 , 23 ] or neural networks [ 17 , 16 ] . Advancements in body part segmentation—assigning anatomical labels to points in a 3D cloud—have unlocked new opportunities in computer graphics, healthcare, and autonomous systems. However, existing methods often struggle in real-world scenarios involving complex poses, occlusions, multi-person interactions, and human-object occlusions. Many approaches rely on synthetic datasets [ 20 , 1 ] , leading to performance degradation when faced with the variability of in-the-wild data [ 22 ] . Furthermore, these methods frequently fail to generalize to unseen body poses and varying sensor noise, limiting their effectiveness in practical applications. To address these limitations, we propose a hybrid framework that merges pose fitting and body part segmentation, allowing each to refine the other. This refinement not only improves pose fitting but also enhances the initial segmentation, enabling further fine-tuning of the segmentation network. We begin with an initial segmentation from the Human3D network [ 26 ] , which provides a coarse assignment of points to body parts and is already fine-tuned on in-the-wild data [ 31 ] . This segmentation serves as the foundation for a two-step optimization procedure that fits the SMPL-X model to the human point clouds. First, body-part centroids guide an approximate alignment of the model pose and orientation [ 6 , 29 ] , establishing a robust initial configuration. Second, the model is refined by considering all points in the cloud, thereby capturing more nuanced pose and shape information [ 23 , 17 ] . Throughout this process, we incorporate a pose prior [ 23 ] to ensure anatomically plausible body configurations, mitigating errors caused by occlusions or missing data. After fitting, we reassign body parts to the 3D point cloud via majority voting over nearest neighbors [ 12 , 24 ] , yielding a segmentation more accurate than the initial prediction. By uniting part segmentation and mesh fitting, our approach more effectively generalizes to diverse environments and remains robust under challenging conditions. We evaluate our approach on three complex datasets – InterCap [ 14 ] , EgoBody [ 31 ] , and BEHAVE [ 5 ] – featuring occlusions, multiple interacting subjects, and human-object interactions. Compared to other leading methods [ 11 , 21 , 26 ] , we observe an up to tenfold boost in pose modeling accuracy and an up to 22% gain in segmentation accuracy. Our contributions are summarized as follows: • Segmentation-Based Pose Fitting. A unified approach that integrates human pose fitting with body part segmentation on point clouds, increasing the accuracy and robustness of the fitting process in the presence of noisy point clouds. • Pose Fitting-Enhanced Segmentation. We utilize fitted SMPL-X meshes to refine body-part labels, leading to more precise segmentation. Additionally, we demonstrate how these improved segmentations can be leveraged for self-supervised fine-tuning of a segmentation network. Our work advances the state-of-the-art in 3D human body fitting and segmentation for point clouds, promising more faithful human representations in real-world, unstructured environments. 2 Related Work Estimating human body pose and shape from 3D point clouds is vital in computer vision, with applications in virtual reality, animation, and human-computer interaction. While extensive research has been conducted on fitting parametric human models to 2D images [ 6 , 15 , 17 ] , we focus on methods that directly operate on 3D point cloud data. Point clouds capture detailed geometric information and avoid the ambiguities inherent in 2D projections, making them valuable for precise human modeling. Several approaches have been developed to fit human poses to point clouds. Bhatnagar et al. 2020a introduced IP-Net , which combines implicit representations with parametric models to reconstruct clothed human bodies from partial scans. IP-Net learns a continuous occupancy field representing the human body, allowing for detailed reconstructions even with incomplete data. Wang et al. 2021 proposed PTF , a method that fits SMPL models to point clouds by considering local geometric features. By leveraging local point distributions, PTF improves fitting accuracy in areas with high curvature or fine-grained details. Zuo et al. 2021 presented a self-supervised approach for 3D human motion reconstruction from depth sequences. Their method leverages temporal coherence without requiring ground truth annotations, effectively reconstructing dynamic human motions. Cai et al. 2023 developed PointHPS , a hierarchical point-based network that directly regresses SMPL parameters from point clouds. PointHPS achieves state-of-the-art results using a point-based encoder-decoder architecture. While these methods can effectively reconstruct common poses, they struggle to accurately capture complex or non-standard poses, particularly those involving occlusions or extreme body configurations. Recently, Marin et al. 2025 introduced NICP (Neural ICP) , which bridges classic ICP with learnable shape representations. NICP iteratively refines a neural deformation field at inference time to better align human models with input

Read the complete text