FutrTrack: A Camera-LiDAR Fusion Transformer for 3D Multiple Object Tracking

Martha Teiko Teye 1,2 , Ori Maoz 2 Matthias Rottmann 3 1 University of Wuppertal, Germany 2 Aptiv, Germany 3 Institute of Computer Science, Osnabruck University, Germany [email protected], [email protected], [email protected] · 2025

Read FutrTrack: A Camera-LiDAR Fusion Transformer for 3D Multiple Object Tracking by Martha Teiko Teye 1,2 , Ori Maoz 2 Matthias Rottmann 3 1 University of Wuppertal, Germany 2 Aptiv, Germany 3 Institute of Computer Science, Osnabruck University, Germany [email protected], [email protected], [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19981.txt.

Introduction

Martha Teiko Teye, Ori Maoz, and Matthias Rottmann present FutrTrack, a transformer-based system for tracking many objects in three dimensions by fusing camera and LiDAR data. The framework builds on existing 3D detectors rather than replacing them. A temporal smoother first refines sequences of detected boxes over a moving window, reducing jitter before identities are assigned.

The tracker itself merges box geometry with fused bird's-eye-view features from both sensor types, so it needs no explicit motion model. Query-based propagation carries identities across frames and re-identifies objects through occlusions and viewpoint changes using both geometric and semantic cues. On the nuScenes and KITTI benchmarks the system reached strong accuracy with few identity switches, showing multimodal features markedly improve query-based tracking.

Catalog record

CREATORMartha Teiko Teye 1,2 , Ori Maoz 2 Matthias Rottmann 3 1 University of Wuppertal, Germany 2 Aptiv, Germany 3 Institute of Computer Science, Osnabruck University, Germany [email protected], [email protected], [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19981

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FutrTrack: A Camera-LiDAR Fusion Transformer for 3D Multiple Object Tracking Martha Teiko Teye 1,2 , Ori Maoz 2 Matthias Rottmann 3 1 University of Wuppertal, Germany 2 Aptiv, Germany 3 Institute of Computer Science, Osnabruck University, Germany [email protected], [email protected], [email protected] Abstract We propose FutrTrack, a modular camera-LiDAR multi-object tracking framework that builds on existing 3D detectors by introducing a transformer-based smoother and a fusion-driven tracker. Inspired by query-based tracking frameworks, FutrTrack employs a multi-modal two-stage transformer refinement and tracking pipeline. Our fusion tracker integrates bounding boxes with multi-modal bird’s eye view (BEV) fusion features from multiple cameras and LiDAR without the need for an explicit motion model. The tracker assigns and propagates identities across frames, leveraging both geometry and semantic cues for robust re-identification under occlusion and viewpoint changes. Prior to tracking, we refine sequences of bounding boxes with a temporal smoother over a moving window to refine trajectories, reduce jitter, and improve spatial consistency. Evaluated on nuScenes and KITTI, FutrTrack demonstrates how query-based transformer tracking methods benefit greatly from multi-modal sensor features compared to previous single sensor modality approaches. With an aMOTA of 74.7 on the nuScenes test set, FutrTrack achieves strong performance on 3D MOT benchmarks, significantly reducing identity switches while maintaining competitive accuracy. Our approach offers an efficient framework to improve transformer-based trackers to compete with other neural network based trackers even with limited data and without pre-training. 1 INTRODUCTION Multi-Object Tracking (MOT) aims to simultaneously detect and maintain consistent identities of multiple targets over time, forming a fundamental component of perception systems in areas such as autonomous driving, robotics, and video surveillance [ Dendorfer, 2020 ] . Figure 1: FutrTrack extracts LiDAR and multi-view camera features and tracks them using query-based fusion attention mechanism. It also refines bounding boxes predictions from existing detectors through a smoother model to serve as queries for the fusion-based attention transformer. Traditional 2D MOT methods operate on image sequences, relying heavily on appearance and motion cues to associate detections frame by frame. While these approaches have achieved remarkable progress with advances in deep learning, their performance is often limited by occlusions, depth ambiguity, and the lack of precise spatial understanding inherent in 2D imagery. To address these challenges, research has increasingly shifted towards 3D Multi-Object Tracking [ Pang et al., 2022 , Bewley et al., 2016 ] , which leverages geometric information from sensors such as LiDAR or stereo cameras to provide accurate localization and motion estimation in real-world coordinates. Figure 2: Overall architecture of FutrTrack. Features from camera and LiDAR are extracted using a standard 2D ResNet and voxel-based encoders respectively. The features are projected into BEV to produce fused features. The fused features together with detection queries and track queries are processed by the FutrTrack to produce tracks. By incorporating the third spatial dimension, 3D MOT enables more reliable tracking in dynamic environments, forming a crucial step toward robust perception in autonomous systems. However, these can be noisy, and even methods that attempt full end-to-end tracking [ Wang et al., 2024 , Lee et al., 2025 , Meinhardt et al., 2022 ] find it difficult to optimize because they are tightly coupled to specific detectors making identity assignment a harder downstream task. Most recently, transformer query-based tracking has been used for both object detection and tracking tasks. Transformer query-based features represent scene elements as a set of learnable queries that interact with encoded sensor data through attention mechanisms, allowing the network to model object relationships in a structured way. In the camera domain, query-based frameworks have achieved strong performance for object detection, yet their effectiveness for object tracking remains limited, largely due to difficulties in maintaining temporal consistency across frames. Recent studies show that combining multiple sensing modalities, particularly camera and LiDAR, can significantly enhance query-based object detection by providing complementary spatial and semantic cues [ Chen et al., 2023 , Liu et al., 2023 ] , but this multi-modal fusion paradigm has not been sufficiently explored in the context of transformer-based multi-object tracking. In detection task, these query-based methods have produced state-of-the-art results and proven that fusing LiDAR and camera features in BEV space provides strong semantic and geometric cues for re-identification. However, most existing approaches to query-based tracking have not yet consistently achieved state-of-the-art (SOTA) performance. These query-based tracking methods [ Zeng et al., 2022 , Zhang et al., 2022 ] are also mostly implemented for 2D tracking but this has not been explored enough for 3D tracking. This motivates a modular framework where detection refinement and identity tracking using transformers are treated as complementary stages. We propose FutrTrack, a multi-modal tracker that harnesses a holistic camera-LiDAR fusion approach to provide rich attention queries for track association. FutrTrack makes use of existing object detectors, refines them using a smoother model, and then tracks them using our new multi-modal query-based tracker that relies on multi-modal features extracted from both camera and LiDAR sensors. The query-based tracker. Benchmark validation shows improved average Multi Object Tracking Precision (aMOTP) with smoother trajectories and reduced identity switches on the nuScenes [ Caesar et al., 2020 ] and KITTI [ Geiger et al., 2012 ] datasets. Our main contributions in this paper are as follows: • A multimodal transformer-based tracking framework that extends query-based tracking to jointly reason over camera and LiDAR modalities, enabling robust multi-object tracking in fused 3D space. • A camera–LiDAR projection and fusion module that aligns image features with LiDAR point representations to construct a unified bird’s-eye-view (BEV) embedding optimized for tracking rather than detection. • On nuScenes and KITTI, our method outperforms prior transformer-based trackers, particularly in occlusion scenarios and for small objects. 2 RELATED WORK MOT has seen significant developments in recent years, particularly with the incorporation of deep learning methods [ Hasan et al., 2022a , Hasan et al., 2022b , Erabati and Araujo, 2023 ] . Current MOT research generally falls into one of the following paradigms: Tracking-by-Detection [ Wojke et al., 2017 , Bewley et al., 2016 , Wang et al., 2024 ] , Joint Detection and Tracking [ Meinhardt et al., 2022 , Bergmann et al., 2019 ] , and Tracking-by-Attention [ Zeng et al., 2022 , Zhang et al., 2022 , Ding et al., 2023 ] . Methods in the Joint Detection and Tracking and Tracking-by-Attention categories largely depend on rich object feature extraction, which demands substantial GPU computation and data. Such requirements are difficult to meet with the limited processing capacity of autonomous vehicles, especially in online modes. In addition, these approaches often do not outperform the simpler Tracking-by-Detection strategy in practice. For these reasons, this work focuses on a lightweight transformer Tracking-by-Attention method. Traditional MOT pipelines separate object detection and tracking, i.e. detection first, then data association like in SORT [ Bewley et al., 2016 ] , DeepSORT [ Wojke et al., 2017 ] and extended Kalman filtering (EKF). With transformer-based detectors like DETR [ Carion et al., 2020 ] , end-to-end tracking approaches emerged, using track queries across frames to fuse detection with temporal continuity [ Meinhardt et al., 2022 , Zeng et al., 2022 , Zhang et al., 2023a ] . MOTR [ Zeng et al., 2022 ] adapts that paradigm for image-based queries, embedding spatial queries, and

Read the complete text