SENTRY : SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking
Mohamad Alansari 1, Yonathan Michael 1∗, Hasan AlMarzouqi 1, Muzammal Naseer 1,2, Naoufel Werghi 1, Sajid Javed 1 · 2026
Read SENTRY : SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking by Mohamad Alansari 1, Yonathan Michael 1∗, Hasan AlMarzouqi 1, Muzammal Naseer 1,2, Naoufel Werghi 1, Sajid Javed 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24449.txt.
Introduction
Alansari and colleagues fix how a leading video tracker remembers, arguing its memory writes are where tracking dies. SAM2-based trackers store past masks, typically selecting them by confidence alone. Under occlusion, rapid motion, and distractors, confident but wrong masks get written into memory, and the tracker drifts. Their 2026 module, SENTRY, is a training free, plug and play refine before write layer that validates each candidate update before committing it.
Multiple segmentation hypotheses per frame are aggregated, backtracked into short tracklets, and checked with neighbour aware, cycle consistent matching against recent trajectories. Masks that are temporally and geometrically consistent are favoured. The base architecture stays untouched; only the write policy changes. Plugged into five strong baselines, SENTRY brings consistent gains on nine benchmarks. New zero shot state of the art results land on several major tracking suites. It runs at practical speeds, adding well under a gigabyte of memory.
Catalog record
| CREATOR | Mohamad Alansari 1, Yonathan Michael 1∗, Hasan AlMarzouqi 1, Muzammal Naseer 1,2, Naoufel Werghi 1, Sajid Javed 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24449
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SENTRY : SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking Mohamad Alansari 1, Yonathan Michael 1∗, Hasan AlMarzouqi 1, Muzammal Naseer 1,2, Naoufel Werghi 1, Sajid Javed 1 Abstract We revisit the memory update mechanism in SAM2-based visual object tracking and identify confidence-only mask selection as the dominant cause of drift under occlusion, rapid motion, and distractors. We introduce SENTRY, a training-free, plug-and-play, refine-before-write module that validates each memory update for short-horizon temporal consistency before committing it. SENTRY aggregates diverse segmentation hypotheses per frame, backtracks them into short tracklets, and uses neighbor-aware cycle-consistent matching against recent trajectories to favor temporally and geometrically consistent masks. It leaves the base architecture untouched, replacing confidence-driven writes with consistency-validated ones. For fair evaluation, we re-evaluate major open-source SAM2-based trackers across all available scales and datasets, filling gaps in prior reports. Integrated into five strong baselines, SENTRY delivers consistent gains across nine benchmarks, achieving new zero-shot SOTA on LaSOT, LaSOT ext , GOT-10k, VOT20, VOT22, and DiDi. Despite these checks, the SAM2-L version runs at 32.8 FPS on an A100, and across compatible hosts adds only about 0.4–0.6 GB VRAM. Our results provide the first unified all-scale evaluation of SAM2-based trackers and show that enforcing temporal validity at write time stabilizes memory-augmented tracking without retraining. (a) (b) Figure 1 : SENTRY performs favorably against SAM2-based variants across major tracking benchmarks. We denote SENTRY applied to SAM2, SAMURAI, and DAM4SAM as SENTRY-S2, SENTRY-SR, and SENTRY-D4S. (a) AUC per benchmark; (b) average AUC across [ 23 , 22 , 60 , 47 ] . 1 Introduction Memory-based architectures have become central to visual object tracking (VOT) by maintaining feature histories that help recover from occlusion, motion blur, and appearance changes. Instead of processing frames independently [ 69 , 75 , 2 , 36 , 62 , 8 , 50 , 41 , 67 ] , recent trackers build temporal associations by querying a memory bank of past representations, improving robustness and identity preservation [ 72 , 16 , 78 , 32 , 31 , 33 , 34 ] . SAM2 [ 51 ] follows this trend with streaming memory and multiple mask hypotheses per frame. Its main weakness is confidence-driven mask selection: the highest-confidence mask is chosen without verifying short-term temporal consistency. Under occlusion, fast motion, or distractors, this cue is often unreliable, causing incorrect masks to be written into memory and triggering drift (Fig. 2 (a)). Existing extensions attempt to mitigate this with motion filtering [ 71 ] or distractor checks [ 58 ] , but they still rely heavily on SAM2’s confidence-based mask selection. Motion filters can break under nonlinear trajectories, and heuristic thresholds often fail in cluttered scenes, leading to similar error accumulation (Figs. 2 b,c). To our knowledge, current SAM2-based trackers lack a systematic, trajectory-level mechanism to verify short-term temporal consistency before a memory write . We address this gap with S AM2- E nhanced N eighbor-aware and T emporally R easoned memor Y (SENTRY), a training-free inference-time memory-admission module that reduces incorrect memory writes. SENTRY targets a different interface from prior motion- or distractor-aware extensions: the memory-write decision. Rather than only improving current-frame prediction or score, it verifies which candidate mask should become future evidence in the autoregressive memory stream. SENTRY replaces confidence-driven selection with a refine-before-write mechanism that explicitly tests each hypothesis for short-horizon temporal plausibility. For every candidate, SENTRY (i) propagates it backward over a fixed window, (ii) applies neighbor-aware checks that penalize identity swaps with nearby instances or distractors, and (iii) evaluates cycle-consistency by requiring that forward–backward trajectories remain aligned with the target’s recent motion. This trajectory-level verification, absent in existing SAM2-style architectures, adds a lightweight temporal reasoning layer without modifying the backbone or memory. Because SENTRY operates entirely outside the backbone and memory module, it integrates cleanly with existing SAM2-based variants and substantially improves robustness when confidence cues are unreliable (see the bottom row in Fig. 2 ). Our contributions are: 1. We propose SENTRY, a training-free refine-before-write module that enforces short-horizon temporal consistency before any memory update in SAM2-based trackers. It replaces confidence-driven selection with a unified mechanism that evaluates multi-source mask candidates through backward re-tracking and neighbor-aware trajectory matching. 2. To the best of our knowledge, we conduct the first unified, all-scale re-evaluation of five open-source SAM2-based trackers across nine benchmarks and four model sizes, establishing consistent and reproducible baselines. Experiments across nine benchmarks including LaSOT [ 23 ] , LaSOT ext [ 22 ] , TNL2K [ 60 ] , GOT-10k [ 28 ] , TrackingNet [ 47 ] , VOT20/22 [ 32 , 31 ] , VOTS24 [ 34 ] , and DiDi [ 58 ] , demonstrate that integrating SENTRY into strong baselines yields consistent improvements and new zero-shot SOTA performance (Fig. 1 ), while preserving real-time operation (Sec. 4.4 , App. 0.J ). (a) (b) (c) Figure 2: Qualitative comparison under occlusion, abrupt motion, and distractor interference. We visualize decoder attention to highlight spatial focus during tracking. (a) SAM2 [ 51 ] loses target identity after occlusion, (b) SAMURAI [ 71 ] drifts under rapid motion, and (c) DAM4SAM [ 58 ] misidentifies distractors due to heuristic filtering. The last row shows SENTRY maintaining accurate localization and consistent masks. 2 Related Work 2.1 Memory-Based Visual Object Tracking Memory networks are widely used in VOT and video object segmentation (VOS) to maintain temporal coherence under appearance change, occlusion, and motion. Key–value memory designs such as STM [ 49 ] and FEELVOS [ 59 ] established long-term feature propagation, later refined through kernelized or decomposed attention [ 55 , 17 , 73 ] and transformer-based memory structures [ 72 , 16 , 78 ] . These ideas were further explored in segmentation-based tracking frameworks such as STMTrack [ 25 ] , and inspired memory-augmented single- and multi-object trackers including RTracker [ 29 ] , MeMOT [ 6 ] , and multimodal variants like MemVLT [ 24 ] . Despite architectural diversity, memory-based VOS trackers typically update memory using frame-level confidence, similarity metrics, or simple schedules. When predictions are unreliable due to occlusion, motion blur, or distractors, incorrect masks or features may enter memory and cause drift. Existing works explore restricted or hierarchical updates [ 16 , 78 ] , but they still assume the selected representation is temporally reliable. Explicit short-horizon verification of a mask’s temporal consistency before committing it to memory has received little attention. 2.2 SAM2 and Its Extensions SAM2 [ 51 ] generalizes memory-based reasoning to a promptable segmentation-tracking framework but retains a confidence-driven memory update, making it vulnerable to high-confidence but temporally inconsistent masks. Extensions stabilize this process from different directions: motion-based priors in SAMURAI [ 71 ] , distractor filtering in DAM4SAM [ 58 ] , long-term memory tree in SAM2Long [ 21 ] , and improved scoring or hierarchical features in SAMITE [ 66 ] and HiM2SAM [ 10 ] . However, they do not explicitly validate competing candidate masks through short-horizon backtracking before the selected prediction becomes future tracking state. Across SAM2-based variants [ 51 , 71 , 21 , 58 , 66 , 10 ] , temporal consistency is treated mainly as a current-frame prediction, scoring, or trajectory-selection problem rather than as an explicit memory-admission