Seg2Track++: Probabilistic Track Validation and Data Association for Multi-Object Tracking and Segmentation

D. Mendonça, T. Barros, C. Premebida, U.J. Nunes · 2026

Read Seg2Track++: Probabilistic Track Validation and Data Association for Multi-Object Tracking and Segmentation by D. Mendonça, T. Barros, C. Premebida, U.J. Nunes (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03875.txt.

Introduction

Mendonça and co-authors validate tracks probabilistically for multi-object tracking and segmentation. Autonomous systems need stable identities and precise masks in moving scenes. Foundation models like SAM2 segment well with no training. Their direct use for tracking suffers from weak association and false positives that spread. Seg2Track plus plus joins SAM2 segmentation with a new track management module.

Tracks link by mask centroid distance and confidence-aware cost modulation. A Bernoulli filter validates each track's existence. Ghost tracks are suppressed before they pollute the output. KITTI MOTS experiments show better identity preservation. False-positive spread falls, and track handling stays robust. No fine-tuning is needed for the gains. Perception teams get a zero-shot tracking layer with principled validation. The filter treats track existence as a probability, not a guess.

Catalog record

CREATORD. Mendonça, T. Barros, C. Premebida, U.J. Nunes
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03875

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seg2Track++: Probabilistic Track Validation and Data Association for Multi-Object Tracking and Segmentation D. Mendonça, T. Barros, C. Premebida, U.J. Nunes Abstract Autonomous systems require robust Multi-Object Tracking and Segmentation (MOTS) to operate reliably in dynamic environments, ensuring consistent object identities and precise mask-level delineation. Foundation models such as SAM2 have shown strong zero-shot generalization for segmentation, but their direct application to MOTS is limited by unreliable track association and false-positive propagation. This work introduces Seg2Track++, a framework that integrates instance segmentation with SAM2 and a novel track management module to perform zero-shot MOTS with enhanced temporal consistency. Tracks are associated using Mask Centroid Distance (MCD) and Confidence-Aware Cost Modulation (CCM), while Probabilistic Track Validation (PTV) employs a Bernoulli filter to validate track existence and suppress ghost tracks. Experimental results on KITTI MOTS demonstrate improved identity preservation, reduced false-positive propagation, and robust track management without fine-tuning. I INTRODUCTION Multi-Object Tracking and Segmentation (MOTS) is a critical perceptual capability for autonomous systems operating in complex, dynamic environments, requiring both the consistent preservation of object identities over time and accurate instance-level spatial segmentation [ 22 ] . Unlike conventional multi-object tracking (MOT), which primarily associates bounding-box trajectories, MOTS requires consistent instance masks under challenging conditions such as occlusions, appearance variation, motion blur, and dynamic backgrounds. Modern MOT pipelines often follow the tracking-by-detection (TBD) paradigm, separating object detection from temporal association [ 20 ] . While this modular design enables scalability, it also introduces vulnerabilities, including association errors and spurious track initialization, especially in crowded scenes or under prolonged occlusion [ 24 ] . Transformer-based foundation models for vision, such as Segment Anything Model 2 (SAM2) [ 15 ] , have greatly advanced instance segmentation [ 7 ] . When integrated into MOTS pipelines [ 4 ] , SAM2’s rich appearance embeddings enhance identity preservation. However, erroneous segmentations may still be instantiated as valid tracks and persist across frames [ 11 ] , leading to false-positive propagation that degrades detection accuracy. False-positive propagation remains a major source of failure in both MOT and MOTS systems [ 17 ] . Spurious detections or segmentation masks can be temporally reinforced, persisting across frames and degrading association quality through identity switches, track fragmentation, and erroneous scene interpretation [ 2 ] . These effects are especially problematic in safety-critical and long-term autonomous applications, highlighting the need for tracking approaches that better account for uncertainty and temporal consistency. This work presents Seg2Track++, a zero-shot MOTS pipeline that addresses persistent challenges in segmentation-centric tracking, including false-positive propagation and association errors. The framework integrates enhanced data association strategies and a temporal probabilistic track validation mechanism, enabling more reliable track management and early suppression of spurious tracks. Evaluation on the KITTI Multi-Object Tracking and Segmentation (KITTI MOTS) benchmark demonstrates consistent improvements in tracking robustness and identity preservation over the original Seg2Track-SAM2. In more details, the key contributions of this work are the following: • Enhanced data association for mask-based MOTS: Integration of Mask Centroid Distance (MCD) and Confidence-Aware Cost Modulation (CCM) strategies adapted for instance mask representations, improving correspondence accuracy under occlusion, motion ambiguity, and appearance variation. • Probabilistic track validation (PTV): A Bernoulli filter-based mechanism that combines detector and SAM2 mask confidence to compute the posterior probability of track validity, enabling early detection and suppression of ghost tracks. • Comprehensive evaluation on KITTI MOTS: Demonstration of improved tracking robustness, reduced false-positive propagation, and enhanced identity preservation, validating the effectiveness of the proposed temporal probabilistic and association mechanisms in a zero-shot SAM-based MOTS setting. II RELATED WORK False-positive propagation is a pervasive failure mode in multi-object tracking systems, arising from fundamental limitations in data association and track management that have been extensively studied in the tracking literature [ 17 , 28 ] . In crowded scenes or under partial and prolonged occlusions, spurious detections and erroneous associations can be temporally reinforced, allowing incorrect hypotheses to persist and contaminate subsequent matching decisions [ 23 ] . This effect is amplified in MOTS, where high-fidelity instance masks may lend excessive confidence to false tracks once initialized [ 11 ] . Fig. 1 : Illustration of the Seg2Track++, a framework for multi-object tracking and segmentation (MOTS) that integrates an instance segmentation model, SAM2, and a track management module. Seg2Track++ manages object tracks over time, initializing new tracks, reinforcing existing ones, and suppressing ghost tracks through mask-driven association and probabilistic temporal validation. A central weakness identified in earlier association pipelines is the reliance on staged matching strategies that favor high-confidence detections, even when lower-confidence candidates exhibit better localization accuracy or temporal consistency [ 12 ] . Recent works address this issue by adopting unified association schemes that evaluate all candidate assignments jointly, reducing early commitment errors and improving association stability [ 18 ] . These strategies are further strengthened by richer similarity formulations that more effectively integrate motion and appearance cues, including combined distance metrics and representations incorporating object shape, Mahalanobis distance, and soft bounding-box IoU [ 19 ] . Together, these advances improve discrimination between true object hypotheses and spurious detections, forming the basis for more reliable tracking under occlusion and ambiguity [ 18 , 19 ] . Beyond association accuracy, duplicate detections under occlusion can trigger erroneous track initialization, motivating occlusion-aware initialization mechanisms that suppress redundant track births [ 18 ] . In both 2D and 3D tracking scenarios, explicit track validity modeling and observational gating strategies have been introduced to reduce ghost trajectories, particularly for distant or partially occluded objects where detection confidence is unreliable [ 13 ] . Related work explores principled Bayesian formulations of object birth, survival, and death, incorporating measurement-driven adaptive birth processes and detection probability modeling to improve robustness against spurious tracks [ 1 ] . Further improvements are achieved through adaptive confidence handling and robust state estimation [ 25 , 8 ] . Soft detection confidence boosting techniques allow low-confidence detections to contribute to tracking without reintroducing multi-stage association biases, while adaptive similarity thresholds improve robustness when tracklets are sparsely updated. In parallel, recent studies highlight the limitations of assuming precise detection localization in state estimation, noting that unmodeled measurement noise leads to state drift during prolonged occlusions [ 13 , 6 ] . Refined Kalman filtering formulations that explicitly account for localization uncertainty have been shown to improve state stability and recovery performance, contributing to consistent gains in MOTA, HOTA, and IDF1 across benchmark datasets [ 13 ] . III METHODOLOGY The proposed framework addresses MOTS, defined as detecting, segmenting, and tracking multiple objects across a sequence of T frames, {I_{1},I_{2},\dots,I_{T}} , where each frame I_{t}\in\mathbb{R}^{H\times W\times C} has height H , width W , and C channels. The objective is to predict K object trajectories, T={T_{1},\dots,T_{K}} , with each trajectory T_{k}=\{(t,m_{t,k},s_{t,k},c_{t,k},\text{id}_{k})\}_{t\in F_{k}} containing a binary mask m_{t,k}\in\{0,1\}^{H\times W} , a confidence score s_{t,k}\in[0,1] , an occlusion score o_{t,k}\in[0,1] , a class label c_{t,k} , and a unique identity \text{id}_{k} . F_{k}\subseteq{1,\dots,T} denotes the frames in which object k is visible. Seg2Track++, illustrated in 1 , which leverages previous work [ 11 ] , integrates three components: (i) an object proposal model, in this case, an instance segmentation model, (ii) SAM2, responsible for transformer and mask-based tracking using attention mechanisms, and (iii) the Seg2Track++ module, which manages

Read the complete text