ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation

Aditya Iyer, Jack Roberts, Nora Ayanian · 2026

Read ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation by Aditya Iyer, Jack Roberts, Nora Ayanian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15812.txt.

Introduction

Iyer, Roberts, and Ayanian build a modular multi-view tracker with honest uncertainty. Tracking objects across cameras breaks when views shift or occlusion hits. End-to-end learners score well but hide uncertainty and resist new layouts. ModTrack keeps learning only in detection and feature extraction. All fusion and tracking steps run in closed analytic form. Sensors report position plus covariance pairs for each sighting. An identity-aware PHD filter holds identities through misses and crowding. WildTrack scores hit 95.5 IDF1 and 91.4 MOTA.

Catalog record

CREATORAditya Iyer, Jack Roberts, Nora Ayanian
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55354
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15812

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation Aditya Iyer, Jack Roberts, Nora Ayanian Abstract Multi-View Multi-Object Tracking (MV-MOT) aims to localize and maintain consistent identities of objects observed by multiple sensors. This task is challenging, as viewpoint changes and occlusion disrupt identity consistency across views and time. Recent end-to-end approaches address this by jointly learning 2D Bird’s Eye View (BEV) representations and identity associations, achieving high tracking accuracy. However, these methods offer no principled uncertainty accounting and remain tightly coupled to their training configuration, limiting generalization across sensor layouts, modalities, or datasets without retraining. We propose ModTrack, a modular MV-MOT system that matches end-to-end performance while providing cross-modal, sensor-agnostic generalization and traceable uncertainty. ModTrack confines learning methods to just the Detection and Feature Extraction stage of the MV-MOT pipeline, performing all fusion, association, and tracking with closed-form analytical methods. Our design reduces each sensor’s output to calibrated position-covariance pairs ( 𝐳 , R ) (\mathbf{z},R) ; cross-view clustering and precision-weighted fusion then yield unified estimates ( 𝐳 ^ , R ^ ) (\hat{\mathbf{z}},\hat{R}) for identity assignment and temporal tracking. A feedback-coupled, identity-informed Gaussian Mixture Probability Hypothesis Density (GM-PHD) filter with HMM motion modes uses these fused estimates to maintain identities under missed detections and heavy occlusion. ModTrack achieves 95.5 IDF1 and 91.4 MOTA on WildTrack , surpassing all prior modular methods by over 21 points and rivaling the state-of-the-art end-to-end methods while providing deployment flexibility they cannot. Specifically, the same tracker core transfers unchanged to MultiviewX and RadarScenes , with only perception-module replacement required to extend to new domains and sensor modalities. Keywords: Multi-view tracking, Sensor-agnostic fusion, Uncertainty propagation, Random finite sets Figure 1 : ModTrack pipeline. Per-camera detections and appearance features (Stage 1) are projected to BEV with Jacobian-propagated covariance (Stage 2), associated via \chi^{2} graph clustering (Stage 3), and precision-weighted into fused (\hat{\mathbf{z}},\hat{R}) pairs (Stage 4). An identity-informed GM-PHD filter (Stage 5) maintains persistent identities, with track predictions feeding back to guide identity assignment (dashed). Neural modules (blue) are confined to Stage 1; all downstream stages are purely analytical. 1 Introduction Multi-view multi-object tracking (MV-MOT) underpins applications from surveillance to collaborative robotics. This task is challenging because objects observed from different viewpoints vary significantly in appearance, scale, and visibility, while occlusion and missed detections disrupt identity consistency across both views and time. Recent end-to-end approaches [ 21 , 26 , 22 , 23 ] achieve high accuracy by jointly learning Bird’s Eye View (BEV) representations and data association, but they provide no interpretable uncertainty estimates and remain tightly coupled to the sensor configuration seen during training, making adaptation to new sensors or modalities difficult without retraining or architectural modification. Classical random finite set (RFS) filters provide principled multi-target estimation with closed-form covariance propagation. However, the standard PHD filter does not maintain persistent identities [ 14 ] . Labeled extensions [ 25 ] recover identities but require combinatorial hypothesis enumeration, and existing multi-view RFS trackers operate in image-space bounding boxes rather than calibrated BEV coordinates, preventing metric covariance propagation for cross-camera fusion and limiting performance in dense tracking scenarios [ 15 ] . We propose ModTrack, a modular MV-MOT system that rivals end-to-end accuracy while providing interpretable, traceable uncertainty, by confining learning to just the Detection and Feature Extraction stage of the pipeline ( Figure 1 ) and performing all fusion, association, and tracking with closed-form analytical methods. The tracking backend uses the GM-PHD predict-update recursion as the backbone for state estimation and component management, augmented with identity-assignment and lifecycle heuristics [ 24 ] . We use the term identity-informed GM-PHD to distinguish this method from the standard unlabeled recursion, and treat it as a practically motivated adaptation of PHD filtering. ModTrack adapts to new sensor configurations at inference time seamlessly, avoiding the retraining or engineering accomodations typically required by end-to-end approaches. It also extends prior RFS trackers by operating directly in calibrated BEV space with metric covariance propagation, narrowing the accuracy gap to learned methods in high-clutter scenes. Across domains, all sensor measurements are represented as calibrated position-covariance pairs (\mathbf{z},R) , so the same analytical core runs unchanged on WildTrack [ 6 ] (7 real cameras), MultiviewX [ 11 ] (6 synthetic cameras), and RadarScenes [ 20 ] (4 automotive radars), requiring only perception-module replacement. On WildTrack, ModTrack achieves 95.5 IDF1 and 91.4 MOTA, surpassing all prior modular methods by over 21 points and rivaling state-of-the-art end-to-end methods. The contributions of this work are threefold. 1. Identity-informed GM-PHD filter with closed-loop propagation. We augment the GM-PHD filter with persistent identity labels, HMM motion modes, and appearance features. Track predictions feed back as association priors into each PHD update, yielding a +8.9 IDF1 gain over spatial information-only association. 2. Analytical covariance chain. A fully interpretable uncertainty pipeline propagates monocular depth variance through Jacobian BEV projection, \chi^{2} -calibrated graph clustering, and common-mode-aware precision-weighted fusion, with every covariance traceable to physical sensor characteristics. 3. Sensor-agnostic deployment flexibility. The identical tracker core operates unchanged on WildTrack, MultiviewX, and RadarScenes with only perception-module replacement, and we validate robustness to sensor dropout over all \binom{7}{k} camera subsets of WildTrack. 2 Related Work End-to-End and Learned Multi-View Tracking. Early multi-camera systems used single-view trackers [ 7 ] independently per camera and reconciled identities across views using appearance cues and epipolar constraints. Learned re-identification embeddings improved cross-view matching [ 33 , 27 ] . More recent approaches perform joint reasoning directly in BEV. EarlyBird [ 21 ] projects per-camera features to a shared ground plane before joint detection and tracking, while later work introduces hierarchical graph reasoning (MCBLT [ 26 ] ), sparse cross-view transformers (SCFusion [ 23 ] ), view-aware attention mechanisms [ 2 ] , and trajectory-level motion–appearance aggregation (MVTrajecter [ 28 ] , TrackTacular [ 22 ] ). Related formulations treat multi-view association as a learned graph problem (UMPN [ 9 ] ) or support variable camera configurations for BEV detection (MIC-BEV [ 32 ] ). Despite strong performance, these methods typically embed sensor geometry and fusion within learned parameters and do not expose explicit uncertainty estimates. Random Finite Set Tracking. Random finite set (RFS) formulations provide a principled Bayesian framework for multi-target estimation under clutter and missed detections. The GM-PHD filter [ 24 ] offers closed-form multi-target state estimation with linear complexity in the number of mixture components, but does not maintain persistent identity labels. The GLMB filter [ 25 ] restores labeled tracking with full Bayesian data association, yet requires enumerating association hypotheses whose number grows combinatorially. The LMB filter [ 18 ] offers a tractable approximation but, like GLMB, carries only statistical labels without appearance integration or feedback from track state to measurement assignment. MV-GLMB-OC [ 15 ] extends GLMB to MV-MOT in pedestrian settings, incorporating occlusion reasoning across cameras, but operating in image-space bounding boxes and is unable to propagate metric covariances in a unified world frame. While RFS-based methods provide rigorous uncertainty handling, they have not been combined with BEV-based geometric fusion or scalable identity reasoning as used in contemporary learned MV-MOT systems. ModTrack bridges learned BEV-based tracking and classical RFS filtering. By combining precision-weighted multi-view fusion with an RFS-style identity-informed GM-PHD backend, it preserves explicit probabilistic state estimates while remaining competitive with modern

Read the complete text