MCBLT: Multi-Camera Multi-Object 3D Tracking in Long Videos

Yizhou Wang, Tim Meinhardt, Orcun Cetintas, Cheng-Yen Yang, Sameer S. Pusegaonkar, Benjamin Missaoui, Sujit Biswas, Zheng Tang, Laura Leal-Taixé · 2024

Read MCBLT: Multi-Camera Multi-Object 3D Tracking in Long Videos by Yizhou Wang, Tim Meinhardt, Orcun Cetintas, Cheng-Yen Yang, Sameer S. Pusegaonkar, Benjamin Missaoui, Sujit Biswas, Zheng Tang, Laura Leal-Taixé (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00692.txt.

Introduction

Yizhou Wang and co-authors present MCBLT, a framework for tracking many objects in 3D across several cameras, in a 2024 paper. Warehouses, shops, and hospitals need systems that follow people and goods through long videos. Older pipelines chain 2D detection, single-view tracking, and re-identification, and they throw away the depth that many cameras could give.

MCBLT first fuses the views, with camera calibration, into 3D detections seen from above. Hierarchical graph neural networks then link those detections over time into full multi-camera tracks. The method generalizes across scenes and camera layouts and handles long-term matching well. It sets new state-of-the-art results on the AICity and WildTrack benchmarks, marking a step toward reliable indoor awareness.

Catalog record

CREATORYizhou Wang, Tim Meinhardt, Orcun Cetintas, Cheng-Yen Yang, Sameer S. Pusegaonkar, Benjamin Missaoui, Sujit Biswas, Zheng Tang, Laura Leal-Taixé
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00692

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MCBLT: Multi-Camera Multi-Object 3D Tracking in Long Videos Yizhou Wang, Tim Meinhardt, Orcun Cetintas, Cheng-Yen Yang, Sameer S. Pusegaonkar, Benjamin Missaoui, Sujit Biswas, Zheng Tang, Laura Leal-Taixé Abstract Object perception from multi-view cameras is crucial for intelligent systems, particularly in indoor environments, e.g . , warehouses, retail stores, and hospitals. Most traditional multi-target multi-camera (MTMC) detection and tracking methods rely on 2D object detection, single-view multi-object tracking (MOT), and cross-view re-identification (ReID) techniques, without properly handling important 3D information by multi-view image aggregation. In this paper, we propose a 3D object detection and tracking framework, named MCBLT, which first aggregates multi-view images with necessary camera calibration parameters to obtain 3D object detections in bird’s-eye view (BEV). Then, we introduce hierarchical graph neural networks (GNNs) to track these 3D detections in BEV for MTMC tracking results. Unlike existing methods, MCBLT has impressive generalizability across different scenes and diverse camera settings, with exceptional capability for long-term association handling. As a result, our proposed MCBLT establishes a new state-of-the-art on the AICity’24 dataset with 81.22 81.22 HOTA, and on the WildTrack dataset with 95.6 95.6 IDF1. 1 Introduction Detecting and tracking objects across multiple cameras is a crucial problem for 3D environment understanding, particularly in retail or warehouse settings, for various applications, including inventory management, security surveillance, or customer behavior analysis. In these use cases, a typical multi-camera system involves numerous cameras with diverse viewing angles and fields of view (FoV). While some cameras usually have overlapping FoVs, others may not share any common visual space. However, this task presents significant challenges such as occlusions, varying lighting conditions, and the need to maintain consistent object identification across different camera views. Moreover, the integration of 3D spatial information from multiple 2D views requires sophisticated algorithms to handle camera calibration errors and perspective distortions. Addressing these issues is crucial for developing robust multi-camera detection and tracking systems that can reliably operate in complex environments. Figure 1 : Comparison among three types of MTMC tracking methods. (a) conducts 2D detection separately and associates objects among different views by appearance-based ReID; (b) considers geometric constraints as well besides appearance for cross-view association; (c) achieves multi-view association in early stage by feature-level aggregation. Existing MTMC tracking methods can be divided into three categories: (i) late multi-view aggregation, (ii) late multi-view aggregation with geometric projection, and (iii) early multi-view aggregation, as shown in Fig. 1 . Late multi-view aggregation pipelines detect objects in each camera view as 2D bounding boxes, with some methods tracking these 2D detections separately before associating them across cameras using re-identification (ReID) embeddings and spatial constraints. Late multi-view aggregation pipelines with geometric projection pipelines [ 35 , 36 , 8 ] further perform additional projections using camera calibration matrices to achieve spatial association for each camera view into a global coordinate system. Recently, [ 30 ] demonstrated the possibility of early aggregation before any perception steps in each camera view. This method first fuses multi-view image information into a unified 3D space and directly conducts both detection and tracking in this 3D space, which can significantly improve detection quality and avoid association errors due to unreliable spatial alignment among different camera views. However, its detection network is designed for a fixed multi-camera scene, so it is not flexible for different environments, different numbers of cameras, or different camera placements. Therefore, it does not have good scalability for large scenes with a large number of cameras. Furthermore, the tracking of [ 30 ] merely relies on a heuristic Kalman filter, which is prone to drifting and struggles with long-term occlusion handling. In this work, we propose MCBLT , a robust and generalizable method for M ulti- C amera B ird’s-eye-view L ong-term T racking in various complex multi-camera indoor/outdoor public environments. Our proposed approach leverages a unified BEV representation and spatio-temporal transformers to directly infer 3D object bounding boxes. Then, the well-localized 3D detections are backprojected to 2D camera images and matched with 2D detections for optimal 2D appearance feature extraction. This allows us to benefit from both highly accurate 2D and 3D detections for ReID and long-term tracking. For the temporal association of detections, we introduce the first 3D tracking-by-detection approach utilizing hierarchical graph neural networks (GNNs) [ 5 ] . The lightweight hierarchical structure allows us to learn a GNN tracking model that directly predicts associations for up to thousands of frames. Compared to prior 2D GNN tracking solutions [ 5 ] , we scale the ability to bridge occlusion gaps by an order of magnitude. Our model consists of a hierarchy of GNNs that associate 3D detections via geometrical and multi-view ReID embedding features. To track objects across very long sequences, we introduce a novel global tracking layer that replaces handcrafted heuristics required to run the GNN in a sliding window fashion [ 5 ] with a model layer that operates globally over the video sequence and matches incoming objects with past tracks. The global layer requires no additional training and significantly boosts occlusion handling and overall tracking performance. We evaluate MCBLT on a large-scale synthetic indoor MTMC dataset AICity’24 [ 32 ] and a real-world outdoor MTMC dataset WildTrack [ 6 ] . MCBLT achieves SOTA on both datasets, with 81.22 HOTA on AICity’24, and 95.6 IDF1 on WildTrack. Overall, our contributions can be summarized as follows: • The first MTMC method efficiently performs early multi-view image aggregation for 3D perception. • A ReID feature extraction method for MTMC tracking based on a 2D-3D detection association algorithm. • The first hierarchical GNN-based 3D multi-object tracking method and a novel global tracking block that unlocks long-term associations across thousands of frames. • We achieve state-of-the-art performance on both the AICity’24 and WildTrack datasets. 2 Related Works Multi-View Object Detection Object detection from multi-view images is an essential technique for understanding 3D geometric information and handling occlusion. With proper synchronization and calibration, images from multiple viewing angles can be integrated into the same space to accurately learn 3D geometry, e.g . , 3D locations, dimensions, and orientations. To begin with, researchers develop multi-view object detection methods based on some classical approaches, e.g . , conditional random field (CRF) [ 1 , 26 ] , probabilistic modeling [ 27 , 9 ] , and etc . , to aggregate information from multiple views. Afterward, when deep learning became popular, people started introducing multi-view object detection methods based on neural networks. MVDet [ 11 ] introduces an end-to-end method, based on convolutional neural networks, which projects dense 2D image features from cameras to a unified ground plane. After spatial aggregation on the projected features by ground plane convolution, MVDet predicts a pedestrian occupancy map to obtain the final detection results. However, MVDet cannot handle different camera settings, e.g . , different numbers of cameras or placements, so that cannot be generalized easily to diverse environments. BEVFormer [ 18 , 37 ] proposes the spatio-temporal transformer to fuse multi-view features into BEV features, which achieves impressive detection accuracy for autonomous driving related applications. The proposed spatio-temporal transformer is based on deformable attention [ 47 ] and to sample BEV features from 2D image features by grid-based reference points in BEV, which has similar setups with our smart

Read the complete text