Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding
Haoran Zhou, Gim Hee Lee · 2025
Read Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding by Haoran Zhou, Gim Hee Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03601.txt.
Introduction
Videos mix motion and meaning across time. Haoran Zhou and Gim Hee Lee present Motion4D in this 2025 work. Foundation models for 2D vision improved dynamic scene analysis from single-camera videos.
Those models often lack 3D consistency, a basic need for scene geometry and motion. The gap causes spatial misalignment and temporal flickering in complex scenes. Motion4D learns 3D-consistent motion and semantics for 4D scene understanding. Consistent 4D models help robotics, video editing, and virtual reality. 4D understanding joins space and time, the frame robots and editors both need.
Catalog record
| CREATOR | Haoran Zhou, Gim Hee Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-03601
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding Haoran Zhou, Gim Hee Lee Abstract Recent advancements in foundation models for 2D vision have substantially improved the analysis of dynamic scenes from monocular videos. However, despite their strong generalization capabilities, these models often lack 3D consistency, a fundamental requirement for understanding scene geometry and motion, thereby causing severe spatial misalignment and temporal flickering in complex 3D environments. In this paper, we present Motion4D, a novel framework that addresses these challenges by integrating 2D priors from foundation models into a unified 4D Gaussian Splatting representation. Our method features a two-part iterative optimization framework: 1) Sequential optimization, which updates motion and semantic fields in consecutive stages to maintain local consistency, and 2) Global optimization, which jointly refines all attributes for long-term coherence. To enhance motion accuracy, we introduce a 3D confidence map that dynamically adjusts the motion priors, and an adaptive resampling process that inserts new Gaussians into under-represented regions based on per-pixel RGB and semantic errors. Furthermore, we enhance semantic coherence through an iterative refinement process that resolves semantic inconsistencies by alternately optimizing the semantic fields and updating prompts of SAM2. Extensive evaluations demonstrate that our Motion4D significantly outperforms both 2D foundation models and existing 3D-based approaches across diverse scene understanding tasks, including point-based tracking, video object segmentation, and novel view synthesis. Our code is available at https://hrzhou2.github.io/motion4d-web/ . DEVA [ 5 ] SAM2 [ 34 ] Motion4D GT BootsTAPIR [ 9 ] CoTracker3 [ 15 ] Motion4D Query Points Figure 1 : Visualization of segmentation and tracking results. We compare Motion4D with state-of-the-art 2D foundation models, highlighting their lack of 3D consistency. As shown, existing 2D approaches often suffer from temporal flickering (green box) or spatial misalignment (red box). 1 Introduction The joint modeling of geometry, semantics and temporal dynamics of 4D scenes is a fundamental task in computer vision with broad applications in robotics, autonomous driving, augmented reality, etc . A key aspect of this problem is the estimation of motion-related priors, such as semantic masks, 2D point tracks, and depth maps, which serve as essential cues for reconstructing and interpreting complex scenes. However, despite significant progress, ensuring consistency and accuracy in the predictions remains challenging due to occlusions, view variations, and motion ambiguity. Recently, the emergence of the Segment Anything Model (SAM) [ 18 ] has revolutionized 2D visual understanding and established itself as a foundational model for image segmentation. Inspired by its success, many related vision models have been developed. For example, Track Any Point [ 7 , 8 ] for point-based object tracking and Depth Anything [ 47 , 48 ] for monocular depth estimation. Although these models have achieved impressive performance by using large-scale datasets and extensive pre-training, they remain inherently limited in maintaining 3D coherence. In practice, even the state-of-the-art SAM2 [ 34 ] model still suffers from significant inconsistencies across frames, including spatial misalignment, temporal flickering, and boundary artifacts ( cf. Figure 1 ). Such limitations arise from their design, which relies on frame-wise processing and lacks explicit 3D reasoning. One promising solution is to lift 2D vision models for 3D understanding by leveraging explicit 3D representations such as Neural Radiance Fields (NeRFs) [ 26 ] or 3D Gaussian Splatting (3DGS) [ 17 ] . Unfortunately, most existing methods [ 52 , 1 , 37 , 51 , 2 ] are specifically designed for static scenes, where they improve consistency across the scene by incorporating multi-view segmentation results into an optimized 3D representation. Applying 2D models to dynamic environments introduces new challenges, including motion complexity, occlusions, and spatio-temporal alignment. These challenges, which are already problematic in 2D, remain difficult to resolve through direct integration with 3D representations. Another limitation of recent works [ 40 , 14 , 21 ] is that they treat 2D priors separately from dynamic 3D representations: they either learn the feature fields independently of the 3D model or decouple semantic understanding from motion estimation. As a result, the limited modeling of appearance and motion ultimately prevents the estimation of coherent predictions. In this paper, we address the challenges of inconsistent 2D predictions and weak 3D integration by developing a unified dynamic representation that models motion and semantics from casual monocular videos. We introduce Motion4D, a method that refines multiple scene characteristics, including semantic masks, 2D point tracks, and depth estimation, while jointly optimizing a 3D Gaussian Splatting model augmented with semantic and motion fields. Motion4D consists of a two-part iterative optimization framework: 1) Sequential optimization that updates the motion and semantic fields in two consecutive stages within short temporal windows to maintain local consistency. 2) Global optimization that performs a joint optimization of all attributes throughout the sequence to ensure coherence. In the first stage of sequential optimization, we propose iterative motion refinement to enforce 3D motion consistency by correcting accumulated errors across sequences using a tracking loss with learned 3D confidence maps. We further introduce an adaptive resampling module to improve motion consistency by inserting new Gaussians into underrepresented regions identified through per-pixel RGB and semantic errors. In the second stage, we propose an iterative semantic refinement process to iteratively update 2D semantic priors by aligning rendered 3D masks with 2D predictions through bounding boxes and prompt points. We introduce DyCheck-VOS, a new benchmark for evaluating video object segmentation in realistic and dynamic scenes with camera and object motion. Extensive evaluations on benchmark datasets show that our Motion4D demonstrates superior performance across various dynamic scene understanding tasks. Our method significantly outperforms both 2D foundation models and 3D representation-based methods in video object segmentation, point-based tracking, and novel view synthesis. Figure 1 shows an example of our qualitative result compared to existing approaches. Our contributions are summarized as: • We propose Motion4D, a model that integrates 2D priors from foundation models into a dynamic 3D Gaussian Splatting representation to achieve consistent motion and semantic modeling from monocular videos. • We design a two-part iterative optimization framework comprising sequential optimization, which updates motion and semantic fields in consecutive stages to maintain local consistency, and global optimization, which jointly refines all attributes to ensure long-term coherence. • We introduce iterative motion refinement using 3D confidence maps and adaptive resampling to enhance dynamic scene reconstruction, and semantic refinement to correct 2D semantic inconsistencies through iterative updates with SAM2. • Our Motion4D significantly outperforms both 2D foundation models and existing 3D methods in tasks, including video object segmentation, point-based tracking, and novel view synthesis. 2 Related Work 2D vision models for scene understanding. As a fundamental aspect of scene understanding, semantic segmentation remains an essential yet challenging task in computer vision. Recently, the Segment Anything Model (SAM) [ 18 ] was introduced as a promptable segmentation network trained on billions of masks, enabling strong zero-shot segmentation performance on new images. Its successor, SAM2 [ 34 ] , extends this paradigm to video by using a streaming memory mechanism, and builds the largest video segmentation dataset (SA-V) to date. SAM2 serves as a foundation model for segmentation across images and videos, achieving state-of-the-art results in video object segmentation [ 5 , 4 , 30 , 28 ] and offering broad applications across various domains. Inspired by its success, many related vision models have