Diffusion Masked Pretraining for Dynamic Point Cloud
Zhuoyue Zhang, Jihua Zhu, Chaowei Fang, Jian Liu, Ajmal Saeed Mian · 2026
Read Diffusion Masked Pretraining for Dynamic Point Cloud by Zhuoyue Zhang, Jihua Zhu, Chaowei Fang, Jian Liu, Ajmal Saeed Mian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03639.txt.
Introduction
Zhuoyue Zhang and six co-authors introduce Diffusion Masked Pretraining, a self-supervised method for learning from dynamic point clouds, sequences of 3D point sets that move over time. Masked reconstruction dominates the field but leaks ground-truth positions into the decoder. It also supervises motion with single fixed targets, discarding uncertainty about plausible paths.
DiMP applies diffusion modeling to both problems. Masked tube centers are noised and then predicted from visible context, removing the positional leakage. Inter-frame motion is also learned as a noise-prediction task, so the encoder targets the full distribution of plausible motions instead of one average. Broad experiments support the design over masked reconstruction baselines.
Catalog record
| CREATOR | Zhuoyue Zhang, Jihua Zhu, Chaowei Fang, Jian Liu, Ajmal Saeed Mian |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 120 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.03639
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diffusion Masked Pretraining for Dynamic Point Cloud Zhuoyue Zhang, Jihua Zhu, Chaowei Fang, Jian Liu, Ajmal Saeed Mian Abstract Dynamic point cloud pretraining is still dominated by masked reconstruction objectives. However, these objectives inherit two key limitations. Existing methods inject ground-truth tube centers as decoder positional embeddings, causing spatio-temporal positional leakage. Moreover, they supervise inter-frame motion with deterministic proxy targets that systematically discard distributional structure by collapsing multimodal trajectory uncertainty into conditional means. To address these limitations, we propose Di ffusion M asked P retraining ( DiMP ), a unified self-supervised framework for dynamic point clouds. DiMP introduces diffusion modeling into both positional inference and motion learning. It first applies forward diffusion noise only to masked tube centers, then predicts clean centers from visible spatio-temporal context. This removes positional leakage while preserving visible coordinates as clean temporal anchors. DiMP also reformulates point-wise inter-frame displacement supervision as a DDPM noise-prediction objective conditioned on decoded representations. This design drives the encoder to target the full conditional distribution of plausible motions under a variational surrogate, rather than collapsing to a single deterministic estimate. Extensive experiments demonstrate that DiMP consistently improves downstream accuracy over the backbone alone, with absolute gains of 11.21% on offline action segmentation and 13.65% under causally constrained online inference. 1 Introduction Self-supervised pretraining with masked autoencoders (MAE) [ 10 , 20 , 38 , 29 , 14 ] has proven effective for static 3D point cloud representation learning by masking local geometric regions and supervising coordinate reconstruction. Building on this, diffusion-based pretraining methods [ 44 , 16 , 36 ] integrate DDPMs [ 11 ] with masked autoencoding, and Point-MaDi [ 36 ] further identifies that injecting ground-truth patch centers as decoder positional embeddings leaks spatial priors [ 24 , 33 , 2 ] , addressing this via diffusion noise over patch centers. Figure 1 : Motivation of DiMP. (a) Deterministic models collapse multimodal motion distributions to a single conditional mean, which lies in a low-probability region when trajectories are multimodal. DiMP targets the full conditional distribution under a variational surrogate. (b) Directly injecting ground-truth masked tube center coordinates as decoder positional embeddings causes positional leakage, whereas DiMP applies forward diffusion noise to masked centers and replaces ground-truth positions with denoised predictions, yielding a leakage-free positional embedding. (c) Qualitative comparisons of denoised inter-frame displacement predictions against the ground truth show that DiMP recovers motion patterns faithfully across representative diffusion timesteps. Detailed analysis of why the model performs well at small and large t is provided in the Appendix G . (d) Qualitative reconstruction results show that DiMP recovers the overall human pose and preserves local geometric structures with high fidelity, yielding reconstructions that closely match the ground truth. Despite this progress, existing methods remain confined to static point clouds. Research has shifted toward dynamic point clouds [ 30 ] . Such dynamic sequences couple spatial layout within each frame with temporal deformation across frames instead of treating scans as unrelated static clouds. Dynamic point cloud understanding is critical to autonomous driving, embodied intelligence, and action recognition. Masked reconstruction methods such as MaST-Pre [ 25 ] and M2PSC [ 9 ] extend the spatio-temporal tube masking paradigm to dynamic sequences. However, they still rely on deterministic proxy targets to supervise inter-frame motion. Accordingly, motion supervision is reduced to fitting conditional expectations. This formulation cannot target the full conditional distribution of motion trajectories. The same spatio-temporal context can correspond to multiple plausible continuations, but mean regression collapses these modes into a single deterministic solution, systematically discarding the distributional structure that is critical for discriminating action classes [ 8 , 23 ] . Additionally, these methods inherit positional leakage from the static domain by injecting ground-truth spatio-temporal tube centers as decoder positional embeddings. In this paper, we propose Di ffusion M asked P retraining ( DiMP ), the first framework to harness the full distributional modeling capacity of DDPMs for dynamic point cloud pretraining. DiMP addresses two intertwined challenges. (1) Spatio-temporal positional leakage: Unlike the static setting where noise can be applied uniformly to all patch centers, visible tube coordinates in videos serve a dual role as geometric anchors for the encoder and as temporal conditioning signals that link consecutive frames into a coherent motion sequence. Corrupting them would undermine motion continuity (Table 5 ), necessitating perturbation exclusively on masked tube centers. (2) Distributional motion modeling: By reformulating inter-frame displacement supervision as a DDPM noise prediction task conditioned on decoded representations, DiMP drives the encoder to internalize the full multimodal distribution of motion trajectories rather than collapsing to deterministic mean estimates. Because different action classes can share nearly identical mean trajectories yet differ sharply in their trajectory variance and shape, capturing this distributional structure is decisive for fine-grained action discrimination. Crucially, these two challenges are not independent, positional leakage is a prerequisite barrier to distributional motion modeling. Leaked ground-truth tube centers collapse decoder cross-attention into local geometric retrieval, severing the informational pathway through which the motion diffusion branch drives the encoder to internalize motion distributions. Resolving positional leakage is therefore necessary before diffusion-based motion uncertainty can be genuinely exploited (see Appendix E ). Concretely, DiMP is built on a masked autoencoding backbone and operates in two stages. In the spatio-temporal center modeling stage, we apply forward diffusion noise only to masked tube centers. The diffusion model is conditioned on visible features and temporal positional encodings to predict the masked center distribution. This design removes positional leakage while preserving clean temporal context. Based on the predicted centers, a masked decoder further fuses visible encoder features with predicted center embeddings to form complete spatio-temporal representations. These representations are supervised by per-frame Chamfer distance to reconstruct the full dynamic point cloud. In the motion-aware modeling stage, we reformulate point-wise inter-frame displacement supervision as a standard DDPM noise prediction task conditioned on the decoded representation. This enables DiMP to genuinely exploit the distributional capacity of DDPMs in dynamic 3D pretraining. Our main contributions are as follows: • We provide a formal argument that deterministic motion supervision systematically discards distributional structure critical to multimodal trajectory uncertainty, and further identify positional leakage as a prerequisite barrier to distributional motion modeling. • We propose DiMP , the first diffusion-based pretraining framework for dynamic point clouds, which reformulates inter-frame displacement supervision as a DDPM noise-prediction task to target the full conditional distribution of motion trajectories under a variational surrogate. • We introduce a center diffusion strategy that confines noise injection exclusively to masked tube centers, resolving positional leakage without compromising temporal inference. • Extensive experiments across action segmentation, semantic segmentation, and action recognition demonstrate that DiMP consistently outperforms prior methods on multiple benchmarks, with particularly pronounced gains under causally constrained online inference. 2 Related work Figure 2 : The pipeline of DiMP. Given a dynamic point cloud, spatiotemporal tubes are extracted and randomly masked. Visible tubes are encoded into visible embeddings via P4DConv, while masked tubes are first perturbed by adding noise to their point coordinates and then fed into a linear projection to obtain masked embeddings ( \mathbf{e}^{m}_{\varepsilon} ).In Stage 1, positional embeddings are constructed by combining ground-truth visible centers with noise-injected masked tube relative centers. These positional embeddings, together with visible and masked embeddings, are fed into a dual-stream