OAMVOS:2nd Report for 5th PVUW MOSE Track
Deshui Miao, Xingsen Huang, Yameng Gu, Xiaogang Yu, Xin Li, Ming - Hsuan Yang · 2026
Read OAMVOS:2nd Report for 5th PVUW MOSE Track by Deshui Miao, Xingsen Huang, Yameng Gu, Xiaogang Yu, Xin Li, Ming - Hsuan Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22837.txt.
Introduction
Miao and five colleagues extend a SAM-based dense tracker to survive long occlusions, fast motion, viewpoint changes, and distractors. The weakness is worst for small objects. Their occlusion-aware version of DAM4SAM adds a reliability state machine, branch-based recovery, delayed memory promotion, and selective memory choice. Memory is committed only after a candidate branch is reconfirmed, and the first conditioning frame is preserved. The design keeps easy cases efficient while improving recovery after long gaps.
Catalog record
| CREATOR | Deshui Miao, Xingsen Huang, Yameng Gu, Xiaogang Yu, Xin Li, Ming - Hsuan Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14751 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22837
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OAMVOS:2nd Report for 5th PVUW MOSE Track Deshui Miao, Xingsen Huang, Yameng Gu, Xiaogang Yu, Xin Li, Ming - Hsuan Yang Abstract SAM-based dense trackers provide strong short-term mask propagation but remain fragile under long occlusion, fast motion, viewpoint change, and distractors. The problem is especially severe for small objects, where a few incorrect memory updates can dominate later predictions. This report presents an occlusion- and reappearance-aware extension of DAM4SAM that improves memory control rather than changing the backbone. The method augments the original SAM3 tracker with four ingredients: a reliability-aware tracking state machine, branch-based recovery, delayed DRM promotion, and a selective policy for native SAM3 memory selection. During stable tracking, the model follows the original single-path propagation process. Once confidence drops, the tracker enters an ambiguous or recovery mode, maintains a small set of candidate branches, and commits memory only after a branch is reconfirmed. For small-object disappearance and reappearance, native memory selection is temporarily bypassed so older anchors remain accessible. In addition, the first conditioning frame is explicitly preserved, and the conditioning-memory budget is moderately enlarged to improve long-gap recovery. The resulting design keeps DAM4SAM efficient in easy cases while improving robustness in sequences dominated by occlusion and reappearance. 1 Introduction Tracking-by-segmentation systems based on foundation models have improved prompt flexibility and mask quality, but robustness under long-range temporal variation is still limited 15 ; 12 ; 13 ; 6 . In practice, three failure modes dominate complex video object segmentation: fast motion 2 , distractor-rich scenes 8 ; 6 , and reappearance after long occlusion 7 or strong viewpoint change. These cases are even harder for small objects because mask geometry is unstable, the object pointer becomes noisy, and a small localization error can erase the target from memory 17 ; 6 . CVPR 2026 5th PVUW challenge has three tracks: Complex VOS on MOSEv2 6 , targeting realistic, cluttered scenes with small, occluded, reappearing, and camouflaged objects under adverse conditions; VOS on MOSE 4 , focusing on challenging, long, and diverse videos; and RVOS on MeViS 3 ; 5 , assessing referring video object segmentation with text or audio. DAM4SAM 16 alleviates part of this problem by promoting selected frames into Dynamic Anchor Memory (DRM), allowing informative historical views to act as conditioning anchors. However, the original pipeline is still mostly single-path: predictions are generated, committed, and later reused as memory in a tightly coupled way. Under severe ambiguity, one incorrect commit can contaminate subsequent memory and trigger compounding drift 9 ; 10 ; 11 . We also observed that native SAM3 1 memory selection, while useful in stable tracking, can harm small-object reappearance because it overemphasizes recent low-quality memories from occlusion periods and suppresses older but semantically correct anchors. The central idea of this project is that robust long-term tracking depends more on memory governance than on a new segmentation backbone 18 ; 14 . We therefore keep the original SAM3 tracker and DAM4SAM memory encoder, but add a control layer that explicitly distinguishes stable tracking from uncertain tracking. Stable frames may update reference statistics and promote new DRM entries. Uncertain frames are treated as provisional: they are explored through a small branch pool and are not allowed to directly dominate long-term memory until consistency is restored. The resulting design combines appearance-aware anchors, motion and geometry consistency, branch-based recovery, delayed DRM promotion, conditional use of native SAM3 memory selection, and explicit preservation of the first conditioning frame during attention. The rest of this report formalizes these components. 2 Methods Figure 1 : Pipeline of our methods. 2.1 Overview The method is built on top of the SAM3-based DAM4SAM tracker. Let I_{t} denote frame t , m_{t} the predicted mask, and p_{t}\in\mathbb{R}^{d} the corresponding object pointer. After initialization, each frame is processed in one of three modes: z_{t}\in\{\texttt{stable},\texttt{ambiguous},\texttt{recovery}\}. (1) In the stable mode, the tracker follows the original DAM4SAM path and performs speculative prediction on the current frame. Reliable predictions are committed as non-conditioning outputs and may later be promoted into DRM. Unreliable predictions trigger an uncertain regime, where several candidate branches are propagated independently, and only a reconfirmed branch is allowed to return to the main path. This separation between candidate generation and memory commitment is the key modification. 2.2 Reliability Estimation in Stable Tracking For every tracked frame, the base predictor returns a primary mask, optional alternative masks, objectness logits, and predicted IoU values. We convert these outputs into four interpretable scores. Appearance score. We maintain an anchor bank \mathcal{A}=\{a_{i}\}_{i=1}^{N_{A}} formed from the initialization frame and later verified high-confidence frames. After \ell_{2} normalization, the appearance score is s_{t}^{\text{app}}=\max_{a_{i}\in\mathcal{A}}\frac{\langle p_{t},a_{i}\rangle+1}{2}, (2) which maps cosine similarity into [0,1] . Motion score. Let c_{t} be the center of the predicted mask. Using the last two stable centers, we estimate a constant-velocity prediction \hat{c}_{t}=c_{t-1}+(c_{t-1}-c_{t-2}), (3) and define s_{t}^{\text{mot}}=\exp\!\left(-\frac{\|c_{t}-\hat{c}_{t}\|_{2}}{\tau_{m}}\right). (4) This term penalizes implausible jumps induced by fast motion or distractor switches. Geometry score. Let A_{t}=|m_{t}| denote mask area and \rho_{t} its aspect ratio. Using medians \bar{A} and \bar{\rho} from stable history, we compute r_{t}^{A}=\min\!\left(\frac{A_{t}}{\bar{A}},\frac{\bar{A}}{A_{t}}\right),\quad r_{t}^{\rho}=\min\!\left(\frac{\rho_{t}}{\bar{\rho}},\frac{\bar{\rho}}{\rho_{t}}\right), (5) and combine them as s_{t}^{\text{geo}}=\text{clip}\!\left(0.7\,r_{t}^{A}+0.3\,r_{t}^{\rho},0,1\right). (6) For very small objects, the area term is relaxed to reduce sensitivity to minor mask fluctuations. Candidate margin. If the predictor returns candidate IoU values \{q_{t}^{(k)}\}_{k=1}^{K} , we define \Delta_{t}=q_{t}^{(1)}-q_{t}^{(2)}, (7) where q_{t}^{(1)}\geq q_{t}^{(2)} are the two largest values. These cues are combined with the top IoU q_{t}=q_{t}^{(1)} to determine the tracking mode: z_{t}=\begin{cases}\texttt{recovery},&A_{t}=0\ \text{or}\ q_{t}<\tau_{\text{rec}}\ \text{or}\ s_{t}^{\text{app}}<\tau_{\text{app,rec}},\\[3.00003pt] \texttt{ambiguous},&q_{t}<\tau_{\text{unc}}\ \text{or}\ s_{t}^{\text{app}}<\tau_{\text{app,unc}}\\ &\hskip 11.00008pt\text{or}\ s_{t}^{\text{mot}}<\tau_{\text{mot}}\ \text{or}\ s_{t}^{\text{geo}}<\tau_{\text{geo}}\ \text{or}\ \Delta_{t}<\tau_{\Delta},\\[3.00003pt] \texttt{stable},&\text{otherwise}.\end{cases} (8) This stage is not deciding whether to store a frame in memory; it is deciding whether the current single-path state is trustworthy enough to be committed. 2.3 Branch-Based Recovery Once a frame is marked uncertain, the tracker constructs a branch pool \mathcal{B}_{t}=\{b_{t}^{(1)},\ldots,b_{t}^{(N_{t})}\}, (9) initialized from the current predictor output. A branch may correspond to the primary mask, one of the strongest alternative masks, or an explicit absent-object hypothesis. Each branch owns an independent copy of the inference state, so hypotheses can evolve without polluting the main path. For non-primary masks, the candidate mask is re-injected as a prompt into the predictor to obtain a branch-specific memory state and object pointer. Branch scores accumulate evidence over time rather than relying on a single frame. For branch b at frame t , we define \displaystyle S_{t}(b)= \displaystyle S_{t-1}(b)+\log(\max(q_{t}(b),\varepsilon))+\frac{1}{2}\log(\max(o_{t}(b),\varepsilon)) (10) \displaystyle+\lambda_{a}s_{t}^{\text{app}}(b)+\lambda_{m}s_{t}^{\text{mot}}(b)+\lambda_{g}s_{t}^{\text{geo}}(b) \displaystyle-\lambda_{e}\,\mathbb{1}[A_{t}(b)=0]. where o_{t}(b) is the objectness probability and \varepsilon avoids numerical instability. In our implementation, appearance receives the largest auxiliary weight because identity preservation matters more than local geometry during recovery. After expansion, only the best branch per root identity is kept, and the global pool is truncated to the top few hypotheses. Recovery ends when one branch wins consistently and satisfies a reconfirmation rule. For the generic case, we require several consecutive wins and strong appearance agreement. For reappearance after a missing streak of length at least L_{\text{miss}} , we use a relaxed rule \displaystyle\mathbf{1}_{\mathrm{reconfirm}}= \displaystyle\mathbf{1}[n_{\mathrm{win}}\geq 1]\cdot\mathbf{1}[q_{t}\geq\tau_{\mathrm{rep,iou}}]\cdot\mathbf{1}[s_{t}^{\mathrm{app}}\geq\tau_{\mathrm{rep,app}}] (11) \displaystyle\cdot\mathbf{1}[\Delta_{t}\geq\tau_{\mathrm{rep},\Delta}]\cdot\mathbf{1}[A_{t}>0]. This relaxation is important for small objects, which often return with weak geometry in the first visible frame. 2.4 Anchor Bank and Delayed DRM Promotion The method maintains two complementary long-term structures: an anchor bank and DRM. The anchor bank stores normalized object pointers from the initialization frame