Enhancing Sa2VA for Referent Video Object Segmentation: 2nd Solution for 7th LSVOS RVOS Track

Ran Hong, Feng Lu, Leilei Cao, Youhai Jiang, Fengjie Zhu · 2025

Read Enhancing Sa2VA for Referent Video Object Segmentation: 2nd Solution for 7th LSVOS RVOS Track by Ran Hong, Feng Lu, Leilei Cao, Youhai Jiang, Fengjie Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15546.txt.

Introduction

Ran Hong leads a team that boosts a video cutting tool guided by words. The task is to cut out, in each frame, the objects a text query names. A recent system joins a language model with a strong cutter. The team improve it with no extra training, using two added parts.

One part checks that the named subject and action truly appear in the video, which cuts false hits. The other part picks key frames with care, catching early views and long range context. On a known test set, the boosted system scores well and takes second place in a 2025 contest track. The work shows checks and frame choice can lift a strong base system.

Catalog record

CREATORRan Hong, Feng Lu, Leilei Cao, Youhai Jiang, Fengjie Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15546

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Sa2VA for Referent Video Object Segmentation: 2nd Solution for 7th LSVOS RVOS Track Ran Hong, Feng Lu, Leilei Cao, Youhai Jiang, Fengjie Zhu Abstract Referential Video Object Segmentation (RVOS) aims to segment all objects in a video that match a given natural language description, bridging the gap between vision and language understanding. Recent work, such as Sa2VA, combines Large Language Models (LLMs) with SAM 2, leveraging the strong video reasoning capability of LLMs to guide video segmentation. In this work, we present a training-free framework that substantially improves Sa2VA’s performance on the RVOS task. Our method introduces two key components: (1) a Video-Language Checker that explicitly verifies whether the subject and action described in the query actually appear in the video, thereby reducing false positives; and (2) a Key-Frame Sampler that adaptively selects informative frames to better capture both early object appearances and long-range temporal context. Without any additional training, our approach achieves a 𝒥 & ℱ \mathcal{J}\&\mathcal{F} score of 64.14% on the MeViS test set, ranking 2nd place in the RVOS track of the 7th LSVOS Challenge at ICCV 2025. 1 Introduction Referring Video Object Segmentation (RVOS) aims to segment the target object in all frames of a video based on a natural language description. Given a video and a textual query such as “the man in the red shirt” , the model must produce a sequence of temporally consistent masks for the referred object. Compared to image-level referring segmentation, RVOS is considerably more challenging due to three factors: (1) the target’s appearance may change drastically across frames because of motion, occlusion, and lighting variations; (2) temporal consistency must be preserved to avoid mask flickering and object drifting; and (3) robust vision-language alignment must be maintained throughout the entire video. The importance of RVOS has been underscored by its inclusion as one of the official tracks in the 7th Large-Scale Video Object Segmentation (LSVOS) Challenge. The challenge features three tracks: (1) Complex Video Object Segmentation (MOSEv2) Ding et al. 2025b , which targets challenging scenarios involving frequent object disappearance and reappearance, severe occlusions, small targets, and adverse conditions such as low light, multi-shot sequences, camouflage, and knowledge-dependent reasoning; (2) Classic VOS Ding et al. 2023b , which segments a specific object instance throughout the entire video given only its first-frame mask, using a mixture of MOSEv1 and LVOS datasets; and (3) Referring VOS (RVOS) Ding et al. 2023a ; Ding et al. 2025a , which focuses on segmenting objects according to natural language expressions. Together, these tracks provide a comprehensive benchmark for advancing both instance-specific and language-driven video object segmentation methods. Early RVOS approaches typically followed a two-stage paradigm, first generating candidate regions and then matching them to the textual query. However, such methods incurred high computational cost and were prone to error propagation. Inspired by DETR Carion et al. 2020 , recent work reformulates RVOS as a query-based set prediction problem. MTTR Botach et al. 2022 introduced a transformer-based architecture that jointly encodes visual and linguistic features and directly predicts pixel-level masks end-to-end. Building on this, ReferFormer Wu et al. 2022 explicitly models temporal relations through temporal attention and multimodal cross-attention, greatly improving temporal mask consistency. More recently, Refer-DINO Liang et al. 2025 Cao et al. 2022 leverages stronger feature backbones and query refinement strategies to further enhance multimodal grounding and segmentation accuracy. The rise of foundation models has unlocked new opportunities for RVOS. Large multimodal models (LMMs) such as InternVL Chen et al. 2025 and LLaVA Liu et al. 2023 exhibit strong open-vocabulary reasoning capabilities, providing rich semantic priors to disambiguate complex referring expressions. Likewise, large-scale segmentation foundation models such as SAM and its successor SAM 2 Ravi et al. 2024 demonstrate impressive zero-shot segmentation capabilities. When combined with language grounding, these models can produce high-quality masks with minimal fine-tuning, enabling scalable solutions for few-shot and open-world RVOS scenarios. Despite these advances, RVOS remains difficult in scenarios involving severe occlusion, small targets, or complex natural language queries. Most existing approaches do not explicitly verify whether the described subject and action actually occur in the video, leading to false-positive segmentations when the query is mismatched. Furthermore, uniformly sampling a fixed number of key frames often fails to capture the relevant subject or action, particularly in long videos where the target may appear late. To overcome these challenges, we propose a novel RVOS framework that integrates large multimodal models with segmentation foundation models. Our approach comprises three components: (1) a Video-Language Checker (VLC) that employs a powerful LMM to determine whether the subject and action described in the text appear in the video, thereby reducing unnecessary computation and false positives; (2) a Key-Frame Sampler (KFS) that combines head-frame sampling with uniform sampling to capture both early appearances and global temporal context, while keeping the number of frames manageable to preserve the representation capacity of the SEG token; and (3) an enhanced Sa2VA Yuan et al. 2025 module that leverages the sampled frames and referring expression to learn a discriminative SEG token, which is then used as a prompt for SAM 2 to segment the key frames and propagate the masks across the entire video. Our contributions are summarized as follows: • We introduce a novel RVOS framework that explicitly verifies semantic consistency between the referring expression and the video, effectively reducing false positives when the query does not match the video content. • We propose an adaptive KFS that balances capturing early object appearances with maintaining global temporal coverage, thereby improving SEG token representation and segmentation accuracy. • We enhance the Sa2VA module by learning a more discriminative SEG token from sampled frames and using it as a prompt for SAM 2 to segment key frames and propagate masks across the entire video. • Extensive experiments on the MeViS benchmark demonstrate that our approach achieves state-of-the-art performance, particularly in scenarios where the target appears late or is described by complex action-related expressions. 2 7th LSVOS Challenge Overview The 7th Large-Scale Video Object Segmentation (LSVOS) Challenge consists of three tracks, each targeting different aspects of video object segmentation and providing a comprehensive benchmark for advancing VOS research. 2.1 Complex Video Object Segmentation (MOSEv2) MOSEv2 focuses on video object segmentation in complex real-world scenarios. Videos in this track feature frequent object disappearance and reappearance, severe occlusions, small targets, and additional challenges such as adverse weather (e.g., rain, snow, fog), low-light conditions (e.g., nighttime, underwater), multi-shot sequences, camouflaged objects, non-physical targets (e.g., shadows and reflections), and scenarios requiring external knowledge. The dataset contains 5,024 videos with over 701,976 high-quality masks for 10,074 objects across 200 categories. Benchmarks on representative VOS methods show substantial performance drops compared to simpler datasets, demonstrating the difficulty of real-world scenarios. 2.2 Classic Video Object Segmentation (Classic VOS) Classic VOS represents the standard video object segmentation setting, where a single object instance must be segmented throughout the video, given only its mask in the first frame. The dataset is a combination of MOSEv1 and LVOS, providing a mixture of familiar and slightly more challenging scenarios. This track evaluates methods under more realistic conditions compared to existing benchmarks like DAVIS and YouTube-VOS, which mainly contain salient, isolated objects. 2.3 Referring Video Object Segmentation (RVOS) RVOS aims to segment

Read the complete text