AgentRVOS for MeViS-Text Track of 5th PVUW Challenge: 3rd Method
Deshui Miao, Chao Yang, Chao Tian, Guoqing Zhu, Kai Yang, Zhifan Mo, Xin Li · 2026
Read AgentRVOS for MeViS-Text Track of 5th PVUW Challenge: 3rd Method by Deshui Miao, Chao Yang, Chao Tian, Guoqing Zhu, Kai Yang, Zhifan Mo, Xin Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22836.txt.
Introduction
Miao and six colleagues describe a video object segmentation system built around Sa2VA for a challenge track on referring expressions in video. Sa2VA first proposes a coarse mask trajectory. Planner, partition, scout, and refinement agents then check, search, and improve it, with a critic scoring candidates and a controller repairing weak hypotheses. If the referred object is absent, the system outputs empty masks rather than inventing one.
Catalog record
| CREATOR | Deshui Miao, Chao Yang, Chao Tian, Guoqing Zhu, Kai Yang, Zhifan Mo, Xin Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13082 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22836
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AgentRVOS for MeViS-Text Track of 5th PVUW Challenge: 3rd Method Deshui Miao, Chao Yang, Chao Tian, Guoqing Zhu, Kai Yang, Zhifan Mo, Xin Li Abstract This report describes a Ref-VOS pipeline centered on Sa2VA and organized with explicit agent roles. The key idea is that Sa2VA should provide the first dense semantic hypothesis, while an agent loop decides whether that hypothesis should be accepted, revised, or refined. The pipeline starts with a target-presence judgment stage. If the referred object does not exist in the video, the system directly outputs zero masks. Otherwise, Sa2VA receives the video and referring prompt and produces a coarse mask trajectory over the full video. This trajectory is treated as a semantic prior rather than a final answer. A planner agent decomposes the query, temporal partition agents identify informative blocks, scout agents search for anchor frames, and refinement agents convert reliable Sa2VA masks into boxes and points for SAM3 propagation. A critic scores candidate trajectories, a reflection controller repairs weak hypotheses, and a collaboration controller reconciles multiple agent branches. The result is a Ref-VOS system in which Sa2VA is responsible for dense grounded understanding, while the agent layer handles presence verification, temporal search, confidence-aware revision, and final mask refinement. 1 Introduction Referring video object segmentation (Ref-VOS) 5 ; 11 ; 8 aims to predict frame-wise pixel masks for the object specified by a natural-language expression 4 ; 18 ; 10 ; 2 ; 22 . Although recent foundation models have substantially improved language-conditioned video understanding, solving Ref-VOS in realistic settings remains challenging 17 ; 13 ; 12 . In particular, the task often cannot be reliably completed by a single end-to-end model invocation 5 ; 15 ; 2 . CVPR 2026 5th PVUW challenge has three tracks: Complex VOS on MOSEv2 9 , targeting realistic, cluttered scenes with small, occluded, reappearing, and camouflaged objects under adverse conditions; VOS on MOSE 6 , focusing on challenging, long and diverse videos; and RVOS on MeViS 5 ; 7 , assessing referring video object segmentation with text or audio. Even a strong model such as Sa2VA 20 may exhibit two recurring failure modes. First, it can be compelled to return a mask for a target that is in fact absent from the video. Second, when the referred object is small, visually ambiguous, or only partially visible across time, the predicted masks may become coarse, temporally unstable, or poorly localized. These errors are not merely failures of mask prediction; more fundamentally, they reflect unresolved decisions about target existence, temporal validity, anchor-frame reliability, and prediction confidence. Motivated by this observation, we adopt a design in which Sa2VA 20 serves as the semantic backbone, while the overall Ref-VOS process is governed by an agentic control loop. The role of Sa2VA is to provide an initial language-grounded interpretation of the query over the video and to produce coarse mask hypotheses together with temporal evidence. However, this initial prediction is not treated as universally trustworthy. Instead, the system explicitly reasons about whether the referred target exists, identifies where the initial interpretation is reliable or uncertain, and determines how subsequent refinement should be carried out. This formulation naturally yields a division of labor between semantic grounding and decision-making 21 ; 16 ; 1 ; 19 . Sa2VA 20 is responsible for coarse semantic localization and preliminary mask generation, whereas the agent layer performs higher-level control: it interprets the query, selects informative frames, evaluates the credibility of Sa2VA 20 outputs, and decides when stronger geometric refinement is necessary. For the latter, the agent invokes SAM3 3 as a dedicated mask refinement and propagation module to improve spatial precision and temporal consistency. Figure 1 : Pipeline of our methods. 2 Problem Formulation Given an input video V=\{I_{t}\}_{t=1}^{T} and a natural-language referring expression q , the goal of referring video object segmentation is to predict a sequence of frame-wise binary masks \mathcal{M}=\{m_{t}\}_{t=1}^{T},\qquad m_{t}\in\{0,1\}^{H_{t}\times W_{t}}, (1) where m_{t} denotes the segmentation mask of the target object in frame I_{t} , and H_{t} and W_{t} are the spatial height and width of the t -th frame, respectively. Unlike conventional Ref-VOS formulations that implicitly assume the referred object is always present, we explicitly introduce an existence variable e\in\{0,1\}, (2) where e=1 indicates that the referred target is present in the video, while e=0 denotes that the query does not correspond to any valid target instance. This variable is essential in practice, since large vision-language segmentation models may still produce plausible but incorrect masks even when the target is absent. Accordingly, the overall prediction can be written as \mathcal{M}=\begin{cases}\{\mathbf{0}\}_{t=1}^{T},&e=0,\\ \Phi(V,q),&e=1,\end{cases} (3) where \mathbf{0} denotes an all-zero mask and \Phi(\cdot) represents the proposed multi-stage segmentation pipeline. In our framework, \Phi is built upon a Sa2VA-centered semantic grounding module, followed by agent-guided anchor selection, mask propagation, and ambiguity-aware refinement. 3 Method Our method follows a coarse-to-fine design that decouples semantic grounding, temporal propagation, and decision-level refinement. The key idea is to retain Sa2VA as the primary semantic interpreter of the language query, while introducing lightweight agentic modules to explicitly reason about target presence, anchor reliability, and frame-level ambiguity. The overall pipeline consists of five stages: (1) presence verification, (2) coarse Sa2VA segmentation, (3) anchor extraction from Sa2VA predictions, (4) SAM3-based propagation, and (5) planner-guided conflict resolution. 3.1 Stage 1: Presence Agent The first stage is a Presence Agent , whose sole purpose is to determine whether the target referred to by q actually exists in the video. Formally, the agent predicts the binary decision e=\Psi_{\mathrm{pres}}(V,q), (4) where \Psi_{\mathrm{pres}} denotes the presence judgment module. If e=0 , the system terminates early and outputs zero masks for all frames. This design prevents the downstream segmentation model from hallucinating object masks for non-existent targets, which is a common failure mode in open-ended language-conditioned video understanding. By separating existence reasoning from dense segmentation, the proposed framework reduces false positives and avoids unnecessary refinement on invalid queries. Table 1: Leaderboard results. Our method achieves third place on this challenge. # Participant J&F J F N-acc. T-acc. Final 1 HITsz_Dragon 0.7897 0.7680 0.8115 0.9615 0.9759 0.9091 2 goodx 0.7106 0.6880 0.7332 1.0000 0.9652 0.8919 3 Ours 0.7130 0.6882 0.7378 0.9615 0.9893 0.8879 5 junjie_zheng 0.6837 0.6633 0.7040 0.8846 0.9679 0.8454 6 rookie7777 0.6420 0.6145 0.6695 0.8462 0.9679 0.8187 3.2 Stage 2: Coarse Segmentation When the Presence Agent predicts e=1 , we invoke Sa2VA on the full video and the referring expression to obtain an initial coarse segmentation trajectory: \tilde{\mathcal{M}}=\{\tilde{m}_{t}\}_{t=1}^{T}=\mathrm{Sa2VA}(V,q). (5) Here, \tilde{m}_{t} denotes the coarse mask predicted for frame I_{t} . In our implementation, Sa2VA directly processes the entire video and returns frame-aligned predictions through predict_forward , yielding a full-video segmentation hypothesis without requiring additional frame sampling or external temporal stitching. We emphasize that the Sa2VA output is not treated as a disposable proposal. Instead, it serves as the semantic backbone of the entire pipeline. Since Sa2VA is responsible for grounding the referring expression to the correct visual entity, its outputs provide the most important semantic evidence about what object should be segmented. However, due to limitations in mask precision and temporal stability, these coarse predictions are further refined by later stages rather than being used as the final output directly. 3.3 Stage 3: Anchor Selection The third stage extracts