Leveraging Multi-View Weak Supervision for Occlusion-Aware Multi-Human Parsing

Laura Bragagnolo, Matteo Terreran, Leonardo Barcellona, Stefano Ghidoni · 2025

Read Leveraging Multi-View Weak Supervision for Occlusion-Aware Multi-Human Parsing by Laura Bragagnolo, Matteo Terreran, Leonardo Barcellona, Stefano Ghidoni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10093.txt.

Introduction

Laura Bragagnolo and 3 co-authors improve the parsing of crowded human figures in this 2025 paper. Multi-human parsing labels each body part in an image and assigns it to the right person. Leading methods do well on public data sets but break down when bodies overlap, a common case in real scenes.

The team's insight is that people hidden from one camera angle may stand apart from another. They train models with several views at once, using weak supervision on instances and a loss that forces agreement across views. They also describe a semi-automatic way to build training masks from color, depth, and skeleton data. In heavy overlap tests, the approach lifted parsing scores by a clear margin over the base model.

Catalog record

CREATORLaura Bragagnolo, Matteo Terreran, Leonardo Barcellona, Stefano Ghidoni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-10093

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging Multi-View Weak Supervision for Occlusion-Aware Multi-Human Parsing Laura Bragagnolo, Matteo Terreran, Leonardo Barcellona, Stefano Ghidoni Abstract Multi-human parsing is the task of segmenting human body parts while associating each part to the person it belongs to, combining instance-level and part-level information for fine-grained human understanding. In this work, we demonstrate that, while state-of-the-art approaches achieved notable results on public datasets, they struggle considerably in segmenting people with overlapping bodies. From the intuition that overlapping people may appear separated from a different point of view, we propose a novel training framework exploiting multi-view information to improve multi-human parsing models under occlusions. Our method integrates such knowledge during the training process, introducing a novel approach based on weak supervision on human instances and a multi-view consistency loss. Given the lack of suitable datasets in the literature, we propose a semi-automatic annotation strategy to generate human instance segmentation masks from multi-view RGB+D data and 3D human skeletons. The experiments demonstrate that the approach can achieve up to a 4.20% relative improvement on human parsing over the baseline model in occlusion scenarios. 1 Introduction Multi-human parsing (MHP), also known as instance-level human parsing, extends human body parts segmentation by integrating information about the specific person the part belongs to. It finds many applications in the fields of multimedia and computer vision, such as human pose estimation [ 10 ] and fashion image manipulation [ 8 ] . While current multi-human parsing methods achieve impressive results [ 24 , 4 ] , the task remains highly challenging. One difficulty is the lack of a standardized set of semantic labels across datasets. For example, the PASCAL-Person-Part dataset [ 3 ] considers only 6 body limb classes, while the widely used Crowd Instance-level Human Parsing (CIHP) dataset [ 9 ] takes into account 19 labels, including clothing and accessories. Such granularity has led to the development of domain-specific approaches. But the most critical challenge arises from occlusions, particularly those caused by overlapping bodies [ 4 ] . To quantify the effect of these, in this work, we analyze the impact of varying levels of occlusion on the accuracy of existing multi-human parsing models. We find that current methods struggle to accurately detect body parts when people are occluding each other, an event that scales with the number of people in the scene. Large portions of the body may be entirely missed, and body parts are frequently mislabeled or assigned to the wrong person. In this work, we show that multiple viewpoints offer a chance of effectively recovering information about human shapes: overlapped bodies in one image may be separated in a different viewpoint. Building on this idea, we propose a novel training framework to leverage multi-view information to fine-tune MHP models. Our approach is based on two main elements: (i) single-view loss on shape information for weak supervision and (ii) a multi-view instance and body part consistency loss. The former provides coarse-grained information about human shapes, while the latter ensures coherent identity and part predictions across multiple views. The combination of the two is referred to as Multi-View Instance-Guided Multi-Human Parsing (MVIG-MHP) framework, while Instance-Guided Multi-Human Parsing (IG-MHP) represents the framework comprising of the single-view contributions only. Our frameworks can be applied to any MHP model, trained on a standard MHP dataset, to improve its performance on occlusions. As the literature lacks a multi-view, multi-human instance segmentation dataset that captures occlusions between people, we propose a new semi-automatic annotation pipeline that generates human instance segmentation masks from multi-view RGB+D data and 3D human skeletons. The pipeline is applied to video sequences displaying multiple people from the CMU Panoptic Studio dataset [ 13 ] , resulting in a new dataset named Panoptic-HuIS. Unlike annotations from multi-human parsing ground truth, the dataset only considers human shape, providing coarser labels, making our framework applicable to models pre-trained on different MHP datasets, independent from their body-part label convention and granularity. Figure 1 provides an overview of the proposed approach. The upper section illustrates the multi-view dataset annotation process, generating human instance segmentation masks used in the fine-tuning process of a multi-human parsing model through two proposed stages, combined into the MVIG-MHP framework. To summarize, the main contributions of this paper are: (i) a comprehensive analysis of the impact of occlusions on state-of-the-art MHP models; (ii) Panoptic-HuIS, a new dataset including scenes with many people overlapping, with human instance segmentation annotations; (iii) a training framework exploiting Panoptic-HuIS to enhance the performance of pretrained MHP models in cases of significant overlaps, thanks to a novel multi-view consistency loss; (iv) evaluation demonstrating the effectiveness in improving on the CHIP dataset when people are overlapping. Figure 1: Our approach for occlusion-robust multi-human parsing includes a multi-view dataset annotation pipeline (top), generating human segmentation masks for the fine-tuning process (middle). The inference phase (bottom, in green) shows results before and after applying MVIG-MHP. 2 Related Works Multi-human parsing. Multi-human parsing methods can be categorized into three main classes: bottom-up, top-down, and single-stage approaches. Bottom-up methods first predict pixel-level body-part labels, then group them into human instances. PGN [ 9 ] segments body parts and then uses edge detection for grouping, while M2FP [ 21 ] , building on the Mask2Former architecture [ 4 ] , exploits the hierarchical structure of the human body and self-attention to relate body parts and instances. Top-down methods follow a different approach by first detecting individuals and then segmenting body parts for each human instance. These techniques are further divided into two-stage and one-stage methods. The former relies on a robust pre-trained object detector, such as Mask R-CNN [ 11 ] , to detect human bounding boxes, followed by single-human parsing [ 18 ] . The latter integrates detection and parsing into a single unified framework. One example is Parsing R-CNN [ 23 ] , which improves segmentation quality via a geometric and context encoding module that captures relationships between body parts. RP R-CNN [ 22 ] further improves semantics by integrating a semantically enhanced Feature Pyramid Network (FPN) [ 16 ] . Differently from most top-down approaches, AIParsing [ 24 ] utilizes the anchor-free detection head FCOS [ 20 ] to localize human instances and combines a CNN-based backbone, such as [ 12 ] , with a FPN to extract multi-scale features for parsing. Single-stage models directly predict part labels and instances in one pass, avoiding detection or grouping. RepParser [ 7 ] injects instance-aware keypoints in the human parsing pipeline, to dynamically parse each person. SMP [ 5 ] uses body center features to obtain instance segmentation and generates offsets from the center of the body to the parts to match without any grouping process. Similarly, Uniparser [ 6 ] proposes a unified correlation representation to learn instance and category features within the cosine space. Multi-view human parsing. Multi-view data remains largely unexplored in instance-level human parsing, despite its established use in 3D segmentation tasks [ 17 ] , including 3D body part segmentation [ 19 , 15 ] . In [ 19 ] authors propose a multi-view system to create a semantic 3D representation of the human body robust to occlusions, aggregating segmentation from each camera into a point cloud. Similarly, Li et al. [ 15 ]

Read the complete text