Collaborative Perceiver: Elevating Vision-based 3D Object Detection via Local Density-Aware Spatial Occupancy

Jicheng Yuan, Manh Nguyen Duc, Qian Liu, Manfred Hauswirth, Danh Le Phuoc · 2025

Read Collaborative Perceiver: Elevating Vision-based 3D Object Detection via Local Density-Aware Spatial Occupancy by Jicheng Yuan, Manh Nguyen Duc, Qian Liu, Manfred Hauswirth, Danh Le Phuoc (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21358.txt.

Introduction

Jicheng Yuan and co-authors present Collaborative Perceiver, a vision based system that finds 3D objects for self driving, in work from 2025. Top down views built from cameras are cheap and rich in context. Past methods squash object features into that view and lose scene context such as roads and pavements.

Their framework learns object detection and space occupancy together. Occupancy data carry local density, which helps the model read the physical scene. Shared structure between the two tasks sharpens features for both. The authors build dense occupancy truth data to train the system. The aim is fuller perception of the world around the car.

Catalog record

CREATORJicheng Yuan, Manh Nguyen Duc, Qian Liu, Manfred Hauswirth, Danh Le Phuoc
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-21358

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Collaborative Perceiver: Elevating Vision-based 3D Object Detection via Local Density-Aware Spatial Occupancy Jicheng Yuan, Manh Nguyen Duc, Qian Liu, Manfred Hauswirth, Danh Le Phuoc Abstract Vision-based bird’s-eye-view (BEV) 3D object detection has advanced significantly in autonomous driving by offering cost-effectiveness and rich contextual information. However, existing methods often construct BEV representations by collapsing extracted object features, neglecting intrinsic environmental contexts, such as roads and pavements. This hinders detectors from comprehensively perceiving the characteristics of the physical world. To alleviate this, we introduce a multi-task learning framework, Collaborative Perceiver (CoP), that leverages spatial occupancy as auxiliary information to mine consistent structural and conceptual similarities shared between 3D object detection and occupancy prediction tasks, bridging gaps in spatial representations and feature refinement. To this end, we first propose a pipeline to generate dense occupancy ground truths incorporating local density information (LDO) for reconstructing detailed environmental information. Next, we employ a voxel-height-guided sampling (VHS) strategy to distill fine-grained local features according to distinct object properties. Furthermore, we develop a global-local collaborative feature fusion (CFF) module that seamlessly integrates complementary knowledge between both tasks, thus composing more robust BEV representations. Extensive experiments on the nuScenes benchmark demonstrate that CoP outperforms existing vision-based frameworks, achieving 49.5% mAP and 59.2% NDS on the test set. Code is available at this link . I INTRODUCTION 3D object detection (OD) [ 1 , 2 ] has been a vital component in autonomous driving (AD) and mobile robots. Compared to LiDAR, cameras can provide richer contextual information regarding object semantics and offer the advantage of cost-effectiveness. One practical paradigm utilizes multi-view images as input to identify and localize objects of interest [ 3 , 4 , 5 , 6 , 7 ] . For instance, BEVDet [ 3 ] transforms 2D multi-view image features into 3D space using the Lift-Splat-Shoot (LSS) based view transformation [ 8 ] . Then, these transformed spatial features are collapsed into BEV grids along the height dimension, upon which detection decoders are applied. Building on BEVDet, many studies [ 4 , 5 , 6 ] further extended it with depth supervision [ 4 ] , temporal fusion [ 5 ] , and spatial-temporal fusion [ 6 ] , leading to significant improvements in 3D perception. Fig. 1: Overview of the proposed Co llaborative P erceiver (CoP), which leverages complementary and consistent knowledge from 3D occupancy and 3D object detection to achieve comprehensive scene understanding. Despite the advancements of the aforementioned methods [ 3 , 6 , 4 , 5 ] , there are not without challenges: C1 Existing 3D detectors [ 2 ] struggle to identify objects with unique or irregular geometries, making it difficult to ensure the safety of AD under complex traffic scenarios [ 9 ] . In contrast, 3D occupancy (Occ.) describes the physical world by assigning an occupied probability to each voxel in the 3D space, delivering a geometrically consistent and precise description of dynamic scenes. However, many existing methods [ 10 , 11 ] inherently assume a homogeneous point density during voxelization, potentially overlooking voxel-level point density variations that are essential for representing fine-grained object properties. C2 Traditional BEV-based approaches [ 3 , 5 , 4 , 6 ] collapse multi-view features along the height dimension, constructing flattened BEV features. However, identifying objects with distinct spatial structures in BEV space is challenging if the dimensionality compression step cannot adaptively preserve their spatial properties [ 12 ] , e.g., a traffic cone at a low height versus a truck at a higher one, as demonstrated in Figure 1 . C3 Relying solely on task-specific local or global information hampers the construction of a universal environmental representation, potentially resulting in inconsistent target recognition and suboptimal performance [ 13 ] . Thus, it is crucial to efficiently extract complementary and consistent knowledge from both structured and unstructured elements of the scene, achieving an effective and comprehensive perception of dynamic environments. In this work, we propose Co llaborative P erceivers ( CoP ), a multi-task learning framework that seamlessly integrates complementary global and local cues shared between 3D object detection and occupancy prediction to build robust BEV representations, thereby boosting 3D detection performance. To tackle challenge C1 , CoP employs 3D occupancy prediction as an auxiliary task to capture informative spatial local features and achieve fine-grained geometric consistency. Considering the significant effort often required to obtain dense occupancy supervision and the non-uniform distribution of point clouds during voxelization, we propose a pipeline that automatically generates local density-aware dense occupancy (LDO) ground truths from offline multi-frame LiDAR data, enriching environmental detail. To alleviate challenge C2 , CoP employs a voxel-height-guided sampling (VHS) strategy to extract height-aware local features, aiming to iteratively refine learned latent spatial information across distinct height ranges. Building upon this, we further develop a collaborative feature fusion (CFF) module that leverages the synergistic cues from global and local knowledge throughout the learning process, thereby addressing challenge C3 . By establishing consistent knowledge shared between tasks, CoP aims to understand environmental correspondences from joint perspectives and enhance perception of structural integrity in the physical world. In summary, our contributions are as follows: • We introduce spatial occupancy states as auxiliary information that, together with LDO ground truth, reconstruct fine-grained spatial structures and enrich environmental detail, thereby boosting 3D detection. • We propose a VHS module that leverages vertical positioning information as a prior to capture fine-grained spatial features across diverse object properties. • Building upon VHS, our CFF module further mines structural correspondences, facilitating feature extraction, enhancement, and interaction from combined global and local perspectives, yielding robust BEV representations. • Extensive experiments on nuScenes benchmark [ 14 ] demonstrate that CoP achieves competitive performance against vision-based frameworks, notably with + 2.1% mAP and + 1.1% NDS improvements, consistently enhancing 3D object detection through an effective learning strategy. II Related Work Fig. 2: Overview of the proposed Co nsistent P erceiver (CoP) and 3D occupancy ground truth (GT) generation pipeline. First, multi-view image features are extracted and transformed into the 3D space following the LSS [ 8 ] . Then, global pooling and local sampling are employed to extract coarse global and fine-grained local features guided by the voxel-height distributions. Next, a global-local collaborative feature fusion block with a C2H module enables adaptive feature extraction, enhancement, and interaction between these features. In addition, the offline-generated local-density-aware 3D occupancy GT with informative and detailed scene semantics works as auxiliary supervision. Vision-centric 3D Object Detection aims to identify 3D objects from images captured by single or multiple cameras [ 15 ] . Prior methods [ 16 , 17 , 18 , 19 , 20 ] extend advanced 2D object detection methods [ 21 , 22 ] to the 3D domain by inferring 3D attributes from 2D counterparts, e.g., FCOS3D [ 23 ] capitalizes on the strong spatial correlation between object’s characteristics and their visual representations to discern 3D objects. Subsequently, BEV-based methods [ 24 , 25 , 12 ] have emerged as the mainstream approach. For instance, BEVDet [ 3 ] leverages the Lift-Splat-Shoot (LSS) [ 8 ] view transformation to convert images from surround-view cameras into unified BEV features for full-range detection. Then, BEVDepth [ 4 ] augments BEVDet [ 3 ] with a

Read the complete text