MGPC: Multimodal Network for Generalizable Point Cloud Completion With Modality Dropout and Progressive Decoding

Jiangyuan Liu, Yuhao Zhao, Hongxuan Ma, Zhe Liu, Jian Wang, Wei Zou · 2026

Read MGPC: Multimodal Network for Generalizable Point Cloud Completion With Modality Dropout and Progressive Decoding by Jiangyuan Liu, Yuhao Zhao, Hongxuan Ma, Zhe Liu, Jian Wang, Wei Zou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03660.txt.

Introduction

Jiangyuan Liu and five co-authors present MGPC, a system that completes three-dimensional point clouds of objects seen only in part. Sensors capture partial views because of occlusion and limited angles, and filling in the missing shape helps robots and scanners work with real objects. Earlier methods did well on synthetic tests but generalized poorly to new objects and real scenes.

MGPC fuses point clouds, color images, and text in one architecture. A modality dropout trick, which randomly hides one input type in training, builds robustness, while a transformer fusion module and a progressive generator refine the shape step by step. The team also built MGPC-1M, a dataset with over a thousand categories and a million training pairs. Strong completion across categories matters for warehouses, homes, and other unplanned spaces.

Catalog record

CREATORJiangyuan Liu, Yuhao Zhao, Hongxuan Ma, Zhe Liu, Jian Wang, Wei Zou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03660

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MGPC: Multimodal Network for Generalizable Point Cloud Completion With Modality Dropout and Progressive Decoding Jiangyuan Liu, Yuhao Zhao, Hongxuan Ma, Zhe Liu, Jian Wang, Wei Zou Abstract Point cloud completion aims to recover complete 3D geometry from partial observations caused by limited viewpoints and occlusions. Existing learning-based works, including 3D Convolutional Neural Network (CNN)-based, point-based, and Transformer-based methods, have achieved strong performance on synthetic benchmarks. However, due to the limitations of modality, scalability, and generative capacity, their generalization to novel objects and real-world scenarios remains challenging. In this paper, we propose MGPC, a generalizable multimodal point cloud completion framework that integrates point clouds, RGB images, and text within a unified architecture. MGPC introduces an innovative modality dropout strategy, a Transformer-based fusion module, and a novel progressive generator to improve robustness, scalability, and geometric modeling capability. We further develop an automatic data generation pipeline and construct MGPC-1M, a large-scale benchmark with over 1,000 categories and one million training pairs. Extensive experiments on MGPC-1M and in-the-wild data demonstrate that the proposed method consistently outperforms prior baselines and exhibits strong generalization under real-world conditions. I Introduction As a widely used 3D representation [ 18 ] , point clouds play a crucial role in spatial perception and measurement systems. Despite their popularity, point clouds captured by modern 3D sensors, such as depth cameras and LIDAR scanners, are often incomplete and sparse due to limited viewpoints, occlusions, and inherent sensing uncertainties [ 15 ] . Recovering complete object geometry from partial observations is therefore a fundamental problem and is critical for many downstream applications, including pose estimation [ 44 ] , manipulation [ 22 ] , and industrial inspection [ 46 ] . With the rapid advancement of deep learning, point cloud completion has been increasingly studied in a data-driven fashion. Early approaches process voxelized representations using 3D CNNs [ 6 , 56 , 67 ] , which suffer from high computational cost and limited scalability as resolution increases. Moving beyond voxel grids, PointNet [ 32 ] and its variants [ 33 , 47 , 8 ] inspired subsequent works [ 62 , 45 , 2 , 26 , 63 , 48 ] to directly operate on unordered point sets through symmetric aggregation functions such as max-pooling. More recently, Transformer-based architectures [ 60 , 23 , 53 , 66 ] have been introduced to better capture long-range dependencies within point clouds. Although these methods achieve strong performance on existing synthetic benchmarks, their generalization ability to novel objects and real-world scenarios remains limited [ 39 , 25 ] . As shown in Fig. 1 (b), state-of-the-art models trained on existing datasets often produce incomplete shapes and scattered outliers on a previously unseen real-world object. Fig. 1: (a) Overview of the proposed framework and dataset. Our method encodes point clouds, RGB images, and text using modality-specific encoders, followed by a scalable Transformer with modality dropout, and predicts completed point clouds via a progressive generator with multi-scale outputs. The proposed multimodal benchmark substantially increases both category coverage and paired-sample scale compared to existing datasets. (b) Qualitative comparison on a novel real-world sample. Methods pretrained on prior datasets struggle to generalize, producing incomplete or noisy reconstructions. Through systematic analysis, we identify three key factors that hinder the generalization of existing point cloud completion methods: (1) Limited modality . Most prior works [ 62 , 60 , 66 ] rely solely on partial point clouds as input. However, geometry alone may be insufficient to resolve shape ambiguity, as distinct objects can exhibit similar geometric structures under certain viewpoints. Incorporating complementary modalities, such as RGB images and textual descriptions, provides additional appearance and semantic cues that are beneficial for robust generalization. (2) Limited scalability . Due to the relatively small scale of existing datasets, many methods depend on strong inductive biases, including local operators (e.g., CNNs [ 6 , 56 , 67 ] and KNN-based modules [ 60 , 23 , 53 , 66 ] ), handcrafted shape priors [ 59 , 5 , 12 , 50 ] , or carefully engineered network components [ 3 , 21 , 35 ] , as summarized in the first row of Fig. 1 (a). While effective in low-data regimes, such designs may constrain scalability and reduce the gains achievable as data scale increases [ 27 , 49 , 19 , 51 ] . (3) Limited generative capacity . The generative capacity of the generator largely determines the fidelity of completed point clouds. Many existing methods adopt folding-based generators [ 58 , 62 , 60 ] or MLP-based generators [ 61 , 21 ] , which deform a 2D grid into 3D surfaces or regress coordinates in a one-shot manner. Such parameterizations often have limited expressiveness for modeling complex 3D geometric distributions, which can hinder fine-detail recovery and lead to suboptimal reconstruction quality. To address the above challenges, we propose MGPC , a simple yet effective M ultimodal framework for G eneralizable P oint cloud C ompletion. As illustrated in the second row of Fig. 1 (a), MGPC encodes point clouds, RGB images, and text into modality-specific tokens and fuses them using a scalable attention-based backbone. To reduce overfitting and improve robustness when auxiliary modalities are unavailable at inference time, a novel modality dropout strategy for multimodal conditioning is introduced. Finally, instead of conventional folding-based or MLP-based generators, MGPC adopts a progressive generator with multi-scale supervision to refine geometry in a coarse-to-fine manner and enhance reconstruction fidelity. Moreover, to fully exploit the scalability of the Transformer architectures and further enhance generalization, we develop an automatic data generation pipeline that incorporates a large Vision-Language Model (VLM) for data filtering, along with more realistic data normalization strategies. Based on this pipeline, we construct MGPC-1M , the largest point cloud completion benchmark to date comprising over 1,000 categories and 1 million paired samples. Extensive experiments on this benchmark demonstrate that MGPC consistently outperforms both single-modal and cross-modal competitors by a substantial margin. Additional in-the-wild evaluations on real-world data further validate the strong zero-shot generalization capability of the proposed approach. In summary, our main contributions are as follows: • We propose MGPC, a unified multimodal framework for point cloud completion that jointly leverages point clouds, RGB images, and text to improve generalization. • We introduce modality dropout, together with a Transformer-based fusion module and a progressive generator, to enhance robustness, scalability, and reconstruction fidelity. • We present an automatic data generation pipeline and construct MGPC-1M, a large-scale benchmark with over 1,000 categories and 1 million training samples. • The proposed method achieves state-of-the-art performance on large-scale benchmarks and real-world scenarios, demonstrating strong generalization. II RELATED WORK II-A Single-Modal Point Cloud Completion The majority of existing point cloud completion methods take partial point clouds as the sole input modality [ 62 , 60 , 66 ] . These works can be broadly categorized into 3D CNN-based methods [ 6 , 56 , 67 ] , point-based methods [ 62 , 45 , 2 , 26 , 63 , 48 ] , and Transformer-based methods [ 60 , 23 , 53 , 66 ] . Early approaches [ 6 , 56 , 67 ] voxelize point clouds into regular grids and apply 3D CNNs in a manner analogous to 2D

Read the complete text