VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Prediction

Ziyue Zhu, Shenlong Wang, Jin Xie, Jiang-jiang Liu, Jingdong Wang, Jian Yang · 2025

Read VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Prediction by Ziyue Zhu, Shenlong Wang, Jin Xie, Jiang-jiang Liu, Jingdong Wang, Jian Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05563.txt.

Introduction

Ziyue Zhu and co-authors borrow a graphics trick to supervise self-driving scene understanding. Their 2025 paper, VoxelSplat, addresses camera-based prediction of which space around a car is occupied, and how it flows. Occlusions and lopsided scenes, mostly still, few moving, make the learning hard.

During training, the model renders its three-dimensional guesses as clouds of soft Gaussian blobs and projects them back into the camera view. That rendering acts as a loss, letting two-dimensional labels police the three-dimensional output where the camera can actually see. The trick turns a popular rendering method into a teacher, improving both the occupancy map and the motion field.

Catalog record

CREATORZiyue Zhu, Shenlong Wang, Jin Xie, Jiang-jiang Liu, Jingdong Wang, Jian Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Prediction Ziyue Zhu, Shenlong Wang, Jin Xie, Jiang-jiang Liu, Jingdong Wang, Jian Yang Abstract Recent advancements in camera-based occupancy prediction have focused on the simultaneous prediction of 3D semantics and scene flow, a task that presents significant challenges due to specific difficulties, e.g., occlusions and unbalanced dynamic environments. In this paper, we analyze these challenges and their underlying causes. To address them, we propose a novel regularization framework called VoxelSplat. This framework leverages recent developments in 3D Gaussian Splatting to enhance model performance in two key ways: ( i ) \left(i\right) Enhanced Semantics Supervision through 2D Projection: During training, our method decodes sparse semantic 3D Gaussians from 3D representations and projects them onto the 2D camera view. This provides additional supervision signals in the camera-visible space, allowing 2D labels to improve the learning of 3D semantics. ( i ​ i ) \left(ii\right) Scene Flow Learning: Our framework uses the predicted scene flow to model the motion of Gaussians, and is thus able to learn the scene flow of moving objects in a self-supervised manner using the labels of adjacent frames. Our method can be seamlessly integrated into various existing occupancy models, enhancing performance without increasing inference time. Extensive experiments on benchmark datasets demonstrate the effectiveness of VoxelSplat in improving the accuracy of both semantic occupancy and scene flow estimation. The project page and codes are available at https://zzy816.github.io/VoxelSplat-Demo/ . 1 1 footnotetext: † Corresponding authors: Jian Yang and Jin Xie. 1 Introduction Robust and accurate perception is crucial for self-driving systems. Camera-centric occupancy map perception has become popular in both industry and academia Huang et al. 2023b ; Li et al. 2023b ; Huang et al. 2021 ; Sima et al. 2023 ; Li et al. 2022 ; Pan et al. 2023 ; Wang et al. 2023a ; Zhang et al. 2023b ; Li et al. 2023a ; Huang et al. 2023a due to its low-cost sensors, robustness, generalizability, and seamless integration with motion planning. Joint semantic and flow prediction on occupancy maps shows great potential, as it can handle both semantic and dynamic understanding, which are key elements for safe driving. \begin{overpic}[figures/ts6.pdf] \put(5.0,76.5){\scriptsize{Semantic Gaussians $GS_{t}$}} \put(74.0,76.5){\scriptsize{scene flow}} \put(60.0,-2.4){\scriptsize{updated Gaussians $GS_{t+1}$}} \put(2.0,-3.0){\scriptsize{2D ground truth $t$ \& $t+1$}} \put(43.0,39.0){\scriptsize{update Gaussians' centers}} \put(44.0,11.5){\scriptsize{loss}} \put(42.0,4.0){\scriptsize{rendering}} \put(11.0,39.0){\scriptsize{loss}} \put(22.0,39.0){\scriptsize{rendering}} \end{overpic} Figure 1 : During training, our method additionally predicts 3D Gaussians GS_{t} representing semantic logits of occupied regions in the current frame. We then obtain Gaussians GS_{t+1} for the future frame by updating their centers using the predicted scene flow. By rendering these Gaussians into the camera views of different time stamps, both semantic and scene flow predictions can be supervised using the multi-frame 2D ground truths. Despite advancements, camera-based occupancy perception faces key challenges. First, large portions of the annotated regions Tian et al. 2024 ; Wang et al. 2023a ; Sima et al. 2023 are inaccessible to the camera views due to occlusion . The supervision intended to improve performance on these invisible regions may propagate misleading signals along the camera rays to the image features, potentially causing adverse effects and deteriorating performance rather than enhancing it. Second, voxelized representations, while advantageous for their grid structure and planner integration, struggle with explicit motion modeling . Voxels are suited for implicit Eulerian motion, yet explicit Laplacian motion is more critical for safe planning in self-driving. Finally, most benchmarks and datasets Caesar et al. 2020 ; Wang et al. 2023a suffer from class and speed imbalances , especially for high-speed objects, limiting robustness to rare dynamic events and hindering safe driving progress. To address these challenges, we present VoxelSplat, a novel occupancy perception framework. At the core of our method is a new Gaussian splatting-based training mechanism. Inspired by the recent success of Gaussian splatting in 3D scene modeling and novel view rendering, we incorporate dynamic Gaussians into our voxelized representation as an additional header, enabling our voxels to render semantics and motion to various viewpoints as images, and calculate and minimize rendering losses. Thanks to the explicit nature of Gaussians, we can explicitly model motion as dynamic movement on Gaussian points, rendering them onto virtual image views to receive supervision through differentiable splat rendering. As shown in Fig. 1 , our method models the occupancy semantic field with 3D Gaussians and their motion with predicted scene flows. The semantic field of the next frame is predicted by transforming the Gaussians using the flows. 2D ground truths from adjacent frames provide supervision for both semantics and flows. Importantly, all additional Gaussian splat rendering and supervision occur only during training as extra rendering headers and losses. During inference, we maintain the original voxelized pipeline without any additional overhead, while benefiting from increased accuracy and robustness. We evaluate VoxelSplat across various benchmarks, showing improvements over state-of-the-art methods in both semantic accuracy by 3.6% and flow estimation accuracy by 20.2%. Additionally, we demonstrate the flexibility of VoxelSplat as a training-time plugin that enhances performance across diverse occupancy prediction architectures. We will release the code upon acceptance, providing the community with an effective tool to boost the training of occupancy perception networks. In summary, our contributions include: • We propose a plug-and-play loss framework that utilizes dynamic Gaussian splatting to boost the learning of both occupancy and flow prediction. • Extensive experiments on benchmark datasets demonstrate that our proposed framework significantly improves the performance of both semantic and flow prediction across various occupancy architectures. 2 Related Work 2.1 Camera-based Occupancy Prediction Occupancy prediction Gan et al. 2024 ; Song et al. 2024 ; Ma et al. 2024a ; Wang et al. 2023b ; Ma et al. 2024b ; Peng et al. 2024 ; Zhao et al. 2024b ; Tang et al. 2024 ; Vobecky et al. 2024 ; Zhang et al. 2023a ; Zheng et al. 2025 ; Wu et al. 2024 has demonstrated significant advantages in 3D scene understanding, making it a crucial task in autonomous driving research Huang et al. 2023b ; Huang et al. 2023a ; Li et al. 2022 ; Li et al. 2023b ; Zhang et al. 2023b ; Tian et al. 2024 ; Miao et al. 2023 ; Gan et al. 2024 ; Yu et al. 2023 ; Yan et al. 2022 ; Yan et al. 2024b . Unlike traditional object detection paradigms, occupancy perception offers several benefits: it expresses dense 3D geometry, accurately provides spatial locations for objects beyond predefined categories, and describes the shapes of irregular obstacles. These advantages have led to a surge in research focused on occupancy prediction tasks Tian et al. 2024 ; Wang et al. 2023a ; Tong et al. 2023 , which predict the occupancy status in the region of interest around the ego vehicle from point clouds or images. Recent methods typically divide the space into voxel grids, estimating the occupancy status and semantics of each grid. For example, BEVDet4D Huang et al. 2021 directly predict the occupancy from bev features. SurroundOcc Wei et al. 2023 proposed a surround-view 3D occupancy perception method that uses spatial 2D-3D attention to lift image features into 3D space, and designed a pipeline to convert point clouds to dense occupancy ground truth. Huang et al . Huang

Read the complete text