Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation
Tianming Liang, Haichao Jiang, Yuting Yang, Chaolei Tan, Shuai Li, Wei-Shi Zheng, Jian-Fang Hu · 2025
Read Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation by Tianming Liang, Haichao Jiang, Yuting Yang, Chaolei Tan, Shuai Li, Wei-Shi Zheng, Jian-Fang Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12702.txt.
Introduction
Finding an object named in words and tracing it through a video is a core skill for editing and robotics. Tianming Liang and six co-authors argue in a 2025 paper that current test sets are too easy: most use clips of a few seconds with the object in plain sight. They built Long-RVOS, a benchmark made of much longer videos.
The set holds more than 2,000 videos that average over 60 seconds each. Objects in them get hidden, vanish and return, and survive cuts between shots. Each object carries three kinds of written description, covering looks, motion, and relations in space and time. The authors add two new scores for steadiness over time. Six leading methods struggled badly on the set. Their own baseline, ReferMo, uses motion cues and a local-to-global design to do better.
Catalog record
| CREATOR | Tianming Liang, Haichao Jiang, Yuting Yang, Chaolei Tan, Shuai Li, Wei-Shi Zheng, Jian-Fang Hu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-12702
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation Tianming Liang, Haichao Jiang, Yuting Yang, Chaolei Tan, Shuai Li, Wei-Shi Zheng, Jian-Fang Hu Abstract Referring video object segmentation (RVOS) aims to identify, track and segment the objects in a video based on language descriptions, which has received great attention in recent years. However, existing datasets remain focus on short video clips within several seconds, with salient objects visible in most frames. To advance the task towards more practical scenarios, we introduce Long-RVOS , a large-scale benchmark for long-term referring video object segmentation. Long-RVOS contains 2,000+ videos of an average duration exceeding 60 seconds, covering a variety of objects that undergo occlusion, disappearance-reappearance and shot changing. The objects are manually annotated with three different types of descriptions to individually evaluate the understanding of static attributes, motion patterns and spatiotemporal relationships. Moreover, unlike previous benchmarks that rely solely on the per-frame spatial evaluation, we introduce two new metrics to assess the temporal and spatiotemporal consistency. We benchmark 7 state-of-the-art methods on Long-RVOS. The results show that current approaches struggle severely with the long-video challenges. To address this, we further propose ReferMo, a promising baseline method that integrates motion information to expand the temporal receptive field, and employs a local-to-global architecture to capture both short-term dynamics and long-term dependencies. Despite simplicity, ReferMo achieves significant improvements over current methods in long-term scenarios. We hope that Long-RVOS and our baseline can drive future RVOS research towards tackling more realistic and long videos. 1 Introduction Referring Video Object Segmentation (RVOS) [ 48 , 3 , 9 ] is an emerging task that aims to identify, track and segment the object in the video based on a natural language description. Unlike traditional semi-supervised VOS models that require first-frame masks as the object prompt, RVOS models rely solely on text descriptions to segment the target. Considering its potential applications like video editing, growing efforts have been devoted to this field [ 32 , 35 , 9 , 17 , 27 ] . Recently, the advent of multi-modal large language models [ 29 , 55 , 19 ] and segment anything models [ 23 , 37 ] has further accelerated this progress [ 1 , 51 , 54 , 58 ] . Figure 1 : Duration comparison of current RVOS datasets. The circle size indicates the number of frames. Figure 2 : Examples from Long-RVOS dataset, with frame indices displayed in the upper left, and selected objects masked in orange \blacksquare . Long-RVOS contains extensive long-term videos, where the objects always undergo occlusion, disappearance-reappearance and shot changing. In addition, the objects are annotated with three different types descriptions: Static , Dynamic and Hybrid . Despite these advances, current RVOS datasets [ 38 , 22 , 13 , 9 ] remain limited to short video clips lasting only a few seconds, with target objects clearly visible in most frames. For state-of-the-art (SOTA) methods, in order to capture the target object accurately, it is inevitable to integrate as much spatiotemporal information as possible from the entire video. However, when the video becomes longer, the number of distractors also increases accordingly, making it more challenging to perform sufficient spatiotemporal reasoning and capture the key information. Especially in RVOS, many text descriptions (e.g., “the cat jumping down”) only refer to a brief fragment in the video. In other hand, due to the GPU memory limitation, existing methods [ 52 , 27 , 28 , 45 ] typically sample 4 \sim 8 frames per video for training, but use all the frames during inference. As the video length increases, the gap between training and inference phases may become more pronounced. Despite these concerns, due to the lack of a long-term RVOS benchmark, the exact challenges posed by longer videos remain unclear. Another concern lies in the evaluation metrics. Existing RVOS benchmarks [ 38 , 22 , 13 , 9 ] typically evaluate performance by simply averaging the frame-wise segmentation metrics (e.g., \cal{J}\&\cal{F} ). However, in real-world videos, the target objects do not appear in every frame, due to occlusion and constrained camera views. Therefore, a robust RVOS model should exhibit a sound temporal consistency. This means it should not only segment the target when it is present, but also be able to predict its absence by outputting an empty mask. However, this capability of temporal consistency can not be adequately reflected by existing metrics. To address these gaps, this work proposes Long-RVOS , a large-scale benchmark for long-term video object segmentation. Long-RVOS is the first minute-level dataset in RVOS field, designed to tackle various realistic long-video challenges such as frequent occlusion, disappearance-reappearance and shot changing, as shown in Figure 1 and Figure 2 . Additionally, we introduce two new metrics for better evaluation of temporal consistency: \mathrm{tIoU} , which measures the temporal overlap between predicted and ground-truth mask sequences; and \mathrm{vIoU} , which further measures the spatiotemporal volume overlap between them. We benchmark 7 SOTA methods on Long-RVOS. The results demonstrate that while notable progress has been achieved in existing short-term benchmarks, these SOTA models still significantly struggle in realistic long-term scenarios, in both frame-level segmentation and video-level temporal consistency. To tackle the challenges posed by Long-RVOS, we present a baseline method ReferMo , which integrates additional motion frames to expand the temporal receptive field during training, and employs a local-to-global architecture to perceive both static attributes, short-term dynamics and long-term dependencies. Specifically, ReferMo decomposes each video into a sequence of clips, each consisting of a high-resolution keyframe and multiple low-resolution motion frames. Then, it perceives the static appearance and short-term motion within local video clip, and captures the global target in long-term context via inter-clip interactions. In this way, the temporal receptive field is expanded from multiple frames to multiple clips, but the training cost does not increase significantly. Despite simplicity, ReferMo achieves significant improvements over existing RVOS approaches, serving a promising baseline for long-term referring video object segmentation. To summarize, our contributions are two folds. 1) We build Long-RVOS, the first large-scale long-term RVOS benchmark. In Long-RVOS, we provide explicit description types and introduce new metrics to enable more comprehensive evaluation. 2) We benchmark 7 SOTA approaches on Long-RVOS, and propose a promising baseline ReferMo to address the challenges in long-video scenarios. These contributions establish a foundation for developing more robust RVOS models to handle realistic long videos. Table 1 : Statistical overview of representative RVOS datasets. Long-RVOS features the longest video duration and the most diverse object classes. Besides, Long-RVOS offers explicit text description types for finer-grained evaluation. Dataset Year Videos Mean duration Total duration Mean frames Masks Objects Object classes Text Text type A2D-Sentences [ 13 ] 2018 3,782 4.9s 5.2h 3.2 58k 4,825 6 6,656 ✗ JHMDB-Sentences [ 13 ] 2018 928 1.3s 0.3h 34.3 32k 928 1 928 ✗ Ref-DAVIS17 [ 22 ] 2018 90 2.9s 0.1h 69.0 14k 205 78 1,544 ✗ Refer-YouTube-VOS [ 38 ] 2020 3,978 4.5s 5.0h 27.2 131k 7,451 94 15,009 ✗ MeViS [ 9 ] 2023 2,006 13.2s 7.3h 79.0 443k 8,171 36 28,570 ✗ Long-RVOS (ours) 2025 2,193 60.3s 36.7h 361.7 2.1M 6,703 163 24,689 ✓ 2 Related Works RVOS Benchmarks. Given an object description, RVOS aims to identify, tracking