Contrastive Multi-Modal Hypergraph Reasoning for 3D Crowd Mesh Recovery

Minghao Sun* Chongyang Xu* Yitao Xie Buzhen Huang † \dagger Kun Li · 2026

Read Contrastive Multi-Modal Hypergraph Reasoning for 3D Crowd Mesh Recovery by Minghao Sun* Chongyang Xu* Yitao Xie Buzhen Huang † \dagger Kun Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13854.txt.

Introduction

Minghao Sun and four colleagues propose paired Multi-modal hyper-net thinking for multi-person 3D crowd mesh finding from severe blocks and depth fuzz. They start robust node codes by mixing RGB features, shape priors and block-aware half poses. A shared-shape hyper-net moves beyond paired limits to model higher-order crowd motion. A hyper-net-based paired learning scheme jointly lifts intra-modal split and forces cross-modal right-angles, sharing global context to infer missing data under severe block.

Catalog record

CREATORMinghao Sun* Chongyang Xu* Yitao Xie Buzhen Huang † \dagger Kun Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45433
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13854

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Contrastive Multi-Modal Hypergraph Reasoning for 3D Crowd Mesh Recovery Minghao Sun* Chongyang Xu* Yitao Xie Buzhen Huang † \dagger Kun Li Abstract Multi-person 3D reconstruction is pivotal for real-world interaction analysis, yet remains challenging due to severe occlusions and depth ambiguity. Current approaches typically rely on single-modality inputs, which inherently lack geometric guidance. Furthermore, these methods often reconstruct subjects in isolation, neglecting the collective group context essential for resolving ambiguities in crowded scenes. To address these limitations, we propose Contrastive Multi-modal Hypergraph Reasoning to synergize semantic, geometric, and pose cues for crowd reconstruction. We first initialize robust node representations by combining RGB features, geometric priors, and occlusion-aware incomplete poses. Additionally, we introduce a pelvis depth indicator as a global spatial anchor, aligning visual features with a metric-scale-agnostic depth ordering. Subsequently, we construct a shared-topology hypergraph that moves beyond pairwise constraints to model higher-order crowd dynamics. To improve feature fusion, we design a hypergraph-based contrastive learning scheme that jointly enhances intra-modal discriminability and enforces cross-modal orthogonality. This mechanism enables the network to propagate global context effectively, allowing it to infer missing information even under severe occlusion. Extensive experiments on the Panoptic and GigaCrowd benchmarks confirm that our method achieves new state-of-the-art performance. Code and pre-trained models are available at https://github.com/SunMH-try/CoMHR . 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding author: Buzhen Huang ([email protected]). This work was supported by the Science Fund for Distinguished Young Scholars of Tianjin under Grant 22JCJQJC00040. I Introduction Multi-person 3D reconstruction is a cornerstone task in computer vision, serving for applications in virtual reality [ 29 ] , crowd analysis [ 30 , 9 , 29 ] , and human interaction [ 12 , 11 ] . While impressive progress has been made in reconstructing individuals, real-world scenarios predominantly involve multiple people interacting in crowded environments. Recovering accurate 3D meshes in such settings remains a formidable challenge due to severe inter-person occlusions [ 13 ] , complex spatial arrangements [ 28 ] , and the inherent depth ambiguity [ 32 ] . Current human mesh recovery methods predominantly rely on single-modality inputs, such as RGB images or 2D keypoints [ 2 , 25 ] . While sufficient for isolated individuals, this reliance becomes a bottleneck in dense crowds, where severe depth ambiguities and frequent inter-person occlusions are common. Traditional one-stage methods attempt to handle crowded scenes by estimating all subjects jointly in a single pass [ 27 , 28 , 2 ] , but often fail in large-scale crowds due to spatial conflicts and limited modeling of complex interactions. Hypergraph-based approaches, such as GroupRec [ 9 ] , explicitly model group interactions, yet their dependence on single-modality RGB inputs restricts high-order relational reasoning and geometric constraints, reducing reconstruction accuracy. Some recent works incorporate auxiliary modalities like LiDAR [ 5 ] , Radar [ 4 ] , or trajectory cues [ 29 ] , but typically use shallow fusion schemes that cannot fully exploit cross-modal information. Moreover, instance-centric designs reconstruct each individual independently, neglecting collective group context and high-order cross-subject priors necessary to resolve ambiguities in crowded scenes. Fig. 1: Our method reconstructs spatially consistent human meshes in extremely dense crowd scenes, capturing accurate relative depth ordering. To address these gaps, we propose Co ntrastive M ulti-modal H ypergraph R easoning ( CoMHR ), which exploits diverse multi-modal cues to enforce strong and robust constraints on individual poses and inter-person interactions. Our core insight is to synergize complementary modalities within a high-order topological structure, using contrastive learning to explicitly and effectively align and fuse heterogeneous features. To support this, we leverage robust geometric priors from foundation models. Although these models provide relative rather than metric depth, they serve as powerful pseudo-ground truth for constraining spatial relationships and depth ordering. By fusing robust geometric cues with RGB semantics and pose priors via a hypergraph [ 7 ] , CoMHR captures high-order crowd interactions and relative depth ordering, ensuring robust reconstruction as illustrated in Fig. 1 . Specifically, we first initialize robust node representations by augmenting RGB features with relative geometric priors and occlusion-aware incomplete poses from foundation models [ 32 , 3 ] . Crucially, to align these visual semantics with metric-scale-agnostic depth ordering, we also introduce Pelvis Depth Indicator as a global spatial anchor, explicitly enforcing front-back ordering among individuals. Building upon these enriched representations, we construct a Shared-Topology Hypergraph regularized by a Hypergraph Contrastive Learning strategy designed to cluster individuals with similar actions (intra-modal) and encourage orthogonality between heterogeneous modalities (cross-modal) [ 24 ] to prevent feature contamination. With the feature space optimized, we execute high-order hypergraph reasoning to propagate global context across the shared topology, utilizing collective group dynamics to infer missing cues for occluded subjects. Finally, the refined representations facilitate accurate regression of pose and shape parameters, ensuring robustness under severe occlusion. Our main contributions are summarized as follows: • We propose Multimodal Hypergraph Reasoning framework that synergizes multi-modal features to explicitly model high-order correlations for robust reconstruction. • We introduce Hypergraph Contrastive Learning strategy to enhance intra-modal discriminability and cross-modal complementarity, ensuring feature robustness. • Extensive experiments on public benchmarks demonstrate that our method achieves state-of-the-art performance with superior computational efficiency and robustness. II Related Work II-A Multi-Person Mesh Recovery Recent multi-person mesh recovery addresses depth ambiguity [ 28 ] , complex interactions [ 12 , 11 ] , and occlusion [ 2 ] , typically via top-down [ 23 ] or bottom-up [ 28 , 2 ] strategies. However, most methods rely on single-modality 2D cues ( e . g ., RGB or keypoints) lacking explicit depth, causing fragility in crowds. While radar [ 4 ] and LiDAR [ 5 ] approaches exist, they require expensive hardware and often employ shallow fusion. Furthermore, high-order interaction modeling remains underexplored, as current group dynamics methods still depend on single modalities. To bridge these gaps, our multi-modal hypergraph framework unifies RGB, depth, and pose cues to model high-order dependencies, ensuring robust depth reasoning and mesh recovery in occluded scenes. II-B Hypergraph Learning for Relational Modeling The HGNN framework [ 7 ] introduced hyperedge convolution to encode high-order correlations, a method widely adopted in trajectory prediction [ 31 ] , navigation [ 17 ] , and pose estimation [ 9 ] . In 3D human reconstruction, hypergraphs have evolved from modeling non-local intra-person joint constraints [ 8 ] to capturing global crowd dynamics. Recent works [ 9 ] extend this to multi-person scenarios, using adaptive learning to represent interactions and occlusions. However, these methods remain limited by their reliance on single-modality features, leading to ambiguity in complex crowds. In contrast, our multi-modal contrastive framework leverages complementary cues to enhance robustness, securing resilient performance in highly occluded crowd scenes where single-modality approaches typically fail. Fig. 2: Overview of CoMHR. We begin with (a) Multi-Modal Initialization , which lifts 2D keypoints via pseudo-depth maps and explicitly anchors RGB features with Pelvis Depth to initialize multi-modal features. These cues are fused to construct (b) a Contrastive Hypergraph , where a shared topology is regularized by contrastive learning. This structure enables (c) High-Order Reasoning to propagate collective context, driving precise SMPL regression. III Method In this work, we present a Multimodal Hypergraph Reasoning framework to mitigate occlusion and depth ambiguity

Read the complete text