Occ Prophet : Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Framework

Junliang Chen *, Huaiyuan Xu *, Yi Wang †, Lap-Pui Chau † · 2025

Read Occ Prophet : Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Framework by Junliang Chen *, Huaiyuan Xu *, Yi Wang †, Lap-Pui Chau † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15180.txt.

Introduction

Junliang Chen and co-authors speed up occupancy forecasting for self-driving cars in this 2025 paper. Such forecasting predicts which space around a car will be filled moments ahead, from camera history alone. Heavy compute has kept the method off small on-board systems.

Their frame, OccProphet, splits the job among three light parts: an observer, a forecaster, and a refiner. A triple-attention trick squeezes features from stacked frames at low cost. Across three driving data sets the frame cut compute by well over half and ran more than twice as fast. Accuracy also rose above the best prior system, the authors report.

Catalog record

CREATORJunliang Chen *, Huaiyuan Xu *, Yi Wang †, Lap-Pui Chau †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15180

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Occ Prophet : Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Framework Junliang Chen *, Huaiyuan Xu *, Yi Wang †, Lap-Pui Chau † Abstract Predicting variations in complex traffic environments is crucial for the safety of autonomous driving. Recent advancements in occupancy forecasting have enabled forecasting future 3D occupied status in driving environments by observing historical 2D images. However, high computational demands make occupancy forecasting less efficient during training and inference stages, hindering its feasibility for deployment on edge agents. In this paper, we propose a novel framework, i.e. , OccProphet, to efficiently and effectively learn occupancy forecasting with significantly lower computational requirements while improving forecasting accuracy. OccProphet comprises three lightweight components: Observer, Forecaster, and Refiner. The Observer extracts spatio-temporal features from 3D multi-frame voxels using the proposed Efficient 4D Aggregation with Tripling-Attention Fusion, while the Forecaster and Refiner conditionally predict and refine future occupancy inferences. Experimental results on nuScenes, Lyft-Level5, and nuScenes-Occupancy datasets demonstrate that OccProphet is both training- and inference-friendly. OccProphet reduces 58% ∼ \sim 78% of the computational cost with a 2.6 × \times speedup compared with the state-of-the-art Cam4DOcc. Moreover, it achieves 4% ∼ \sim 18% relatively higher forecasting accuracy. Code and models are publicly available at https://github.com/JLChen-C/OccProphet . 1 1 footnotetext: ∗ Equal contribution † Corresponding author 1 Introduction Figure 1: Illustration of OccProphet. OccProphet only receives multi-camera video input and produces future occupancies. Figure 2: Comparison of performance between Cam4DOcc and OccProphet. Autonomous driving holds significant promise for reshaping transportation and urban mobility. Perceiving 3D surroundings is critical for autonomous vehicles. There are typically two 3D perception paradigms: detection-based perception and occupancy-based perception. The detection-based paradigm, such as monocular or multi-view 3D object detection ( Chen et al., 2016 ; Wang et al., 2021 ; Huang et al., 2021 ; Wang et al., 2022 ) , equips autonomous vehicles with 3D perception capabilities by detecting traffic participants and assigning them 3D bounding boxes. However, due to limitations in pre-defined object categories and rigid detection boxes, the detection-based paradigm struggles to generalize to unknown objects and irregular structures, which are likely to appear in real-world traffic scenarios. To relieve these constraints, the occupancy-based perception paradigm ( Huang et al., 2023a ; Wei et al., 2023b ; Wang et al., 2023b ; Tian et al., 2023 ; Xu et al., 2025 ) offers a more general and fine-grained representation of the environment through learning occupied states in 3D space. This paradigm provides a stronger perception ability to autonomous vehicles, enabling them to better comprehend complex traffic environments. Despite advancements in occupancy perception, most existing methods can only perceive the past and present states of the environment. They lack the ability to capture and understand environmental dynamics and subsequently forecast the future scene. Forecasting is essential for safe planning, which assists autonomous vehicles to avoid potential collisions ( Ding et al., 2019 ; Song et al., 2020 ; Ding et al., 2021 ; Huang et al., 2023b ; Pan et al., 2024 ) . Although some bird’s-eye view (BEV) approaches have achieved object motion forecasting in the environment ( Hu et al., 2021 ; Zhang et al., 2022 ; Fang et al., 2023 ; Ferenczi et al., 2024 ) , their forecasts are restricted on a 2D plane. This 2D forecasting limits the comprehensive understanding of the entire 3D dynamic scene. In light of this limitation, occupancy forecasting methods ( Weng et al., 2021 ; Weng et al., 2022 ; Mersch et al., 2022 ; Khurana et al., 2023 ; Agro et al., 2024 ) shift to predict future 3D occupancy for the whole environment. However, these methods rely on point cloud inputs from expensive LiDAR kits. To explore more cost-effective solutions, Cam4DOcc ( Ma et al., 2024a ) introduces a camera-only benchmark and baseline for occupancy forecasting. While Cam4DOcc achieves remarkable performance in occupancy forecasting compared to its counterparts, the high computational cost makes it less efficient during training and inference. It hampers feasibility of deployment on edge agents, such as autonomous vehicles that operate under restricted computational budgets. In this paper, we propose a novel framework, dubbed as OccProphet (shown in Figure 2 ), to efficiently and effectively perform camera-only occupancy forecasting. In OccProphet, we design three lightweight components to forecast future states: the Observer , Forecaster , and Refiner . The Observer adopts 4D feature aggregation and a tripling-attention fusion strategy on the reduced-resolution features to extract spatio-temporal information efficiently. The Forecaster then infers future states according to the scene condition and the Observer’s outputs. Finally, the Refiner enhances the quality of the forecast results through spatio-temporal interactions. The main contributions of this paper are summarized as follows: • We propose OccProphet, a novel camera-only occupancy forecasting framework, which is both efficient and effective during training and inference, towards on-vehicle deployment. • We design a lightweight Observer-Forecaster-Refiner pipeline for OccProphet. The Observer extracts spatio-temporal features from historical observations; the Forecaster conditionally predicts coarse future states; the Refiner promotes forecasting accuracy. • Experimental results demonstrate that OccProphet achieves higher forecasting accuracy with less than half the computational cost of Cam4DOcc. These improvements are consistently observed across the nuScenes ( Caesar et al., 2020 ) , Lyft-Level5 ( Houston et al., 2021 ) , and nuScenes-Occupancy ( Wang et al., 2023a ) datasets, highlighting the superior efficiency and effectiveness of OccProphet (shown in Figure 2 ). 2 Related Work 2.1 Occupancy Prediction Occupancy prediction aims at modeling the current 3D occupancy layout in space, by observing historical and current environments. Occupancy prediction is adept at providing 3D dense descriptions for complex traffic scenarios, thereby garnering increasing attention from academia and industry. SSCNet ( Song et al., 2017 ) was the first semantic occupancy prediction work, which simultaneously predicted occupied voxels and their semantics for an indoor scene using a depth image. MonoScene ( Cao & De Charette, 2022 ) extends SSCNet to outdoor scenarios by using an RGB image and incorporating stronger supervisions. Training and evaluating occupancy prediction networks require benchmarks with ground truth occupancy labels, which are challenging due to the complexity of densely annotating 3D outdoor driving scenes. Wang et al. ( Wang et al., 2023b ) propose OpenOccupancy, the first large-scale benchmark for semantic occupancy prediction, which covers multiple sensing modalities and provides high-resolution dense occupancy annotations. Occ3d, developed by Tian et al. ( Tian et al., 2023 ) , is another widely used benchmark for occupancy prediction, whose high-quality labels are from the proposed technique of image-guided occupancy label refinement. Considering the inherently dense nature of depicting 3D environments, ideal occupancy perception emphasizes balancing efficiency and accuracy. Although some studies explore sparse queries ( Li et al., 2023b ) or tri-perspective view (TPV) representation ( Huang et al., 2023a ) to elevate the efficiency of occupancy prediction, they inevitably sacrifice fine-grained details of 3D space. In contrast, many other methods ( Zhang et al., 2023 ; Wei et al., 2023b ; Ma et al., 2024b ) utilize 3D feature volumes to preserve 3D details of the scene, leading to higher occupancy prediction accuracy. Recently, COTR ( Ma et al., 2024b ) proposes a compact occupancy representation that preserves geometric details while reducing computational costs. Despite the significant advancements in

Read the complete text