Online 3D Multi-Camera Perception through Robust 2D Tracking and Depth-based Late Aggregation
Vu-Minh Le, Thao-Anh Tran, Duc Huy Do, Xuan Canh Do, Huong Ninh, Hai Tran · 2025
Read Online 3D Multi-Camera Perception through Robust 2D Tracking and Depth-based Late Aggregation by Vu-Minh Le, Thao-Anh Tran, Duc Huy Do, Xuan Canh Do, Huong Ninh, Hai Tran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-09946.txt.
Introduction
Vu-Minh Le and 5 co-authors present, in a 2025 paper, a way to lift ordinary two-dimensional multi-camera tracking into three dimensions. Tracking people across many cameras is key for large-scale surveillance, and depth data plus camera calibration can place each target in real space. Rebuilding a whole tracking system for 3D, though, is often not practical for teams with working 2D setups.
Their method keeps the 2D tracker, then uses depth to rebuild each target as a point cloud and recover its 3D box by clustering and heading correction. An improved matching step uses local identity consistency to assign global identities across frames. On the 2025 AI City Challenge 3D tracking data set, the system placed third on the leaderboard.
Catalog record
| CREATOR | Vu-Minh Le, Thao-Anh Tran, Duc Huy Do, Xuan Canh Do, Huong Ninh, Hai Tran |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-09946
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Online 3D Multi-Camera Perception through Robust 2D Tracking and Depth-based Late Aggregation Vu-Minh Le, Thao-Anh Tran, Duc Huy Do, Xuan Canh Do, Huong Ninh, Hai Tran Abstract Multi-Target Multi-Camera Tracking (MTMC) is an essential computer vision task for automating large-scale surveillance. With camera calibration and depth information, the targets in the scene can be projected into 3D space, offering unparalleled levels of automatic perception of a 3D environment. However, tracking in the 3D space requires replacing all 2D tracking components from the ground up, which may be infeasible for existing MTMC systems. In this paper, we present an approach for extending any online 2D multi-camera tracking system into 3D space by utilizing depth information to reconstruct a target in point-cloud space, and recovering its 3D box through clustering and yaw refinement following tracking. We also introduced an enhanced online data association mechanism that leverages the target’s local ID consistency to assign global IDs across frames. The proposed framework is evaluated on the 2025 AI City Challenge’s 3D MTMC dataset, achieving 3rd place on the leaderboard. 1 Introduction † † footnotetext: * Equal contributions. † † footnotetext: † Corresponding author’s email: [email protected] Multi-Target Multi-Camera Tracking (MTMC), the task of detecting all objects on multiple cameras and assigning consistent IDs to unique entities in the scene, is a crucial computer vision problem for automatic processing of the large amount of data that surveillance cameras produce. In a multi-camera environment, with proper camera calibration and depth information, 3D bounding boxes are excellent representations of the target’s location, providing great detail for 3D scene perception. While conventional MTMC methods have achieved remarkable performance in 2D space [ 58 , 61 , 19 ] , extending this capability to 3D remains a challenging problem. The primary obstacles arise from the need to effectively fuse multi-view information, handle partial occlusions, and represent object-level 3D geometry in a scalable and interpretable way. To resolve this, recent approaches such as EarlyBird and BEV-SUSHI [ 46 , 55 ] opt to fuse multi-view image data or detections into a unified 3D space before tracking. While these methods have been shown to achieve state-of-the-art performance on 3D MTMC tasks, the task of transitioning established 2D MTMC systems to 3D tracking typically requires replacing all 2D components, which is impractical for large-scale surveillance systems due to the complexity and cost of such overhauls. Figure 1 : A depiction of the 3D Multi-Camera Tracking problem, where a target in the space is represented by a single 3D bounding box, and can be projected to the 2D view of all cameras in the scene. The task is to localize the 3D bounding boxes of these targets and assign them consistent IDs as they traverse across the scene. To address this problem, this paper proposes a novel framework that enables Online 3D Multi-Target Multi-Camera Tracking by constructing the target in the point-cloud space with the aid of the 2D MTMC results, then recovering the 3D bounding box in the space. Our framework consists of two main stages. First, we perform 2D multi-camera tracking to obtain 2D bounding boxes and consistent identities across views, where we introduce an enhanced association mechanism for assigning global IDs to targets using their local ID consistency. Then, we fuse depth information from multiple cameras to reconstruct localized point clouds per tracked object. These point clouds are transformed into a global coordinate system using extrinsic calibration parameters, creating a 3D representation of the object. A bounding box around the object is then created and refined to yield the final tracking result. Our contributions in this paper are as follows: • We propose a framework that flexibly extends the traditional online 2D MTMC systems to output 3D tracking results through late aggregation. • We introduce an enhanced online association mechanism that improves ID assignment performance on cross-camera tracking by utilizing each target’s local ID as a feature to propagate the global ID across frames. • We present a mechanism for extending the ID-assigned 2D bounding boxes into the 3D space by leveraging segmentation and point-cloud clustering on the depth maps to create 3D bounding boxes. • Our proposed methods are validated on the 2025 AI City Challenge 3D MTMC dataset [ 45 ] , achieving third place on the track’s public leaderboard. 2 Related Works 2.1 Object Detection Object detection aims to localize and classify all relevant objects in RGB images, which is a crucial step in the majority of Multi-Target Tracking algorithms. Recent methods fall primarily into two categories: CNN-based and Transformer-based approaches. CNN-based methods are further divided into anchor-based and anchor-free approaches. Anchor-based detectors [ 12 , 40 , 13 , 3 ] rely on predefined bounding boxes (anchors) for predicting object locations, achieving high accuracy, although at the expense of flexibility for objects of unconventional scales and aspect ratios. Anchor-free CNN methods [ 48 , 11 , 17 , 68 ] predict object locations directly without anchors, demonstrating improved real-time efficiency and robustness across diverse scales. Transformer-based detectors [ 5 , 69 , 62 , 33 , 26 , 70 ] have emerged as a promising upgrade, employing self-attention mechanisms that effectively capture long-range contextual dependencies, leading to enhanced localization and classification performance, especially on small and similar objects, albeit requiring extra computational overhead. 2.2 Multi-Target Multi-Camera Tracking In recent years, Multi-Camera Tracking has seen great developments. Most 2D approaches typically start by tracking on individual cameras first, before aggregating the results from all cameras and synchronizing the IDs for the same entity in the scene [ 15 ] . During single-camera tracking, the generated bounding boxes from the object detectors first have their features extracted, such as appearance features [ 65 , 25 , 56 ] , motion [ 4 , 34 , 1 ] , and pose information [ 19 , 58 , 61 ] . The extracted information is then used to facilitate the association of the bounding boxes between multiple entities in different frames using various heuristics [ 1 , 56 , 64 , 47 , 4 , 34 ] or learnable methods [ 55 , 35 , 8 , 42 ] . These methods are categorized into the tracking-by-detection paradigm. Afterwards, Multi-Camera Tracking makes use of the extracted appearance features to synchronize the IDs between cameras for all unique targets in the scene [ 60 , 36 , 67 , 14 , 43 ] . In a multi-view setting, camera calibration can be leveraged to further enhance association accuracy, through homography matrices [ 19 , 9 , 44 , 59 ] , or epipolar geometric affinity [ 58 ] . The cross-camera matching problem is then solved through various optimization frameworks, with the two most common being view-by-view Hungarian matching [ 19 , 14 ] and hierarchical clustering [ 36 , 44 , 9 ] . Online approaches pose the additional issue of ID assignment and management, as the algorithm may only utilize information from past frames to assign IDs to incoming detections, as opposed to the full trajectories that offline methods have access to. Recent approaches utilize the Hungarian algorithm to match clusters to existing tracks based on appearance and spatial features [ 19 , 18 ] , doing away with the