ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations

Tianming Liang Kun-Yu Lin Chaolei Tan Jianguo Zhang Wei-Shi Zheng Jian-Fang Hu · 2025

Read ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations by Tianming Liang Kun-Yu Lin Chaolei Tan Jianguo Zhang Wei-Shi Zheng Jian-Fang Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14607.txt.

Introduction

Tianming Liang and five colleagues present ReferDINO, a model that cuts out a named object through a whole video. The task, referring video object segmentation, takes a text phrase and must paint the matching object in every frame. It blends language sense, fine pixel work, and time tracking, a hard trio.

The model inherits its word-region alignment from strong visual grounding systems. A deformable decoder then grows the mask step by step, guided by predicted locations. A temporal module feeds time-aware text features between frames so the target stays consistent. A pruning trick drops weak queries to keep speed up. On five benchmarks it beat prior methods while running in real time.

Catalog record

CREATORTianming Liang Kun-Yu Lin Chaolei Tan Jianguo Zhang Wei-Shi Zheng Jian-Fang Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14607

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations Tianming Liang Kun-Yu Lin Chaolei Tan Jianguo Zhang Wei-Shi Zheng Jian-Fang Hu Abstract Referring video object segmentation (RVOS) aims to segment target objects throughout a video based on a text description. Despite notable progress in recent years, current RVOS models remain struggle to handle complicated object descriptions due to their limited video-language understanding. To address this limitation, we present ReferDINO , an end-to-end RVOS model that inherits strong vision-language understanding from the pretrained visual grounding foundation models, and is further endowed with effective temporal understanding and object segmentation capabilities. In ReferDINO, we contribute three technical innovations for effectively adapting the foundation models to RVOS: 1) an object-consistent temporal enhancer that capitalizes on the pretrained object-text representations to enhance temporal understanding and object consistency; 2) a grounding-guided deformable mask decoder that integrates text and grounding conditions to generate accurate object masks; 3) a confidence-aware query pruning strategy that significantly improves the object decoding efficiency without compromising performance. We conduct extensive experiments on five public RVOS benchmarks to demonstrate that our proposed ReferDINO outperforms state-of-the-art methods significantly. Project page: https://isee-laboratory.github.io/ReferDINO . 1 Introduction Referring Video Object Segmentation (RVOS) Botach et al. 2022 ; Wu et al. 2022 aims to segment the target object in a video, referred by a given text description. This emerging task is potentially benefitial for many interactive video applications, and has therefore attracted great attention in the computer vision community. Compared to the unimodal video segmentation tasks Perazzi et al. 2016 ; Xu et al. 2018 , RVOS is more challenging since it requires a strong vision-language capability to understand complicated descriptions and associate visual objects with text. Figure 1 : (a) The SOTA method Luo et al. 2023 fails to handle the descriptions involving composite object attributes and spatial locations. (b) GroundingDINO Liu et al. 2024 can neither understand the temporal dynamics nor perform pixel-wise segmentation. Despite significant progress in recent years, existing RVOS models Botach et al. 2022 ; Wu et al. 2022 ; Luo et al. 2023 ; Miao et al. 2023 ; Yuan et al. 2024 ; He and Ding 2024 still suffer from many common problems. For example, they often struggle to handle the descriptions involving complicated object attributes and spatial locations, as shown in Figure 1 (a). These problems are mainly caused by the insufficient vision-language capabilities of current models, which, in turn, result from the limited scale and diversity of available RVOS data. In this work, we propose ReferDINO , an end-to-end RVOS approach that effectively addresses the above problems by inheriting strong vision-language understanding and spatial grounding capabilities from foundational visual grounding models. Represented by GroundingDINO Liu et al. 2024 , these foundation models benefit from extensive image-text pretraining and showcase strong potential in understanding complex object-text associations. The critical challenges in adapting visual grounding models to the RVOS task are the lack of temporal understanding and pixel-wise segmentation capabilities, as illustrated in Figure 1 (b). To bridge these gaps and enable efficient adaptation to RVOS, we propose three key innovations in ReferDINO. First , we introduce an object-consistent temporal enhancer that leverages cross-modal text representations from foundational models to facilitate cross-frame object interaction, improving both temporal understanding and object consistency. Second , we elaborate a grounding-guided deformable mask decoder , which integrates both text and grounding conditions through efficient attention mechanisms to achieve accurate object segmentation. Third , we design a confidence-aware query pruning strategy to alleviate the computational bottlenecks without compromising performance, enabling our ReferDINO to satisfy the efficiency demands of video tasks. These designs fully unleash the spatial grounding knowledge of pretrained foundation models and endow ReferDINO with effective temporal understanding and object segmentation capabilities. As shown in Figure 1 , our ReferDINO overcomes the limitations of state-of-the-art RVOS models and the visual grounding models, enabling accurate referring video object segmentation. Extensive experiments on five public RVOS benchmarks empirically show remarkable improvements of ReferDINO over state-of-the-art (SOTA) methods. For example, on Ref-DAVIS17 dataset, ReferDINO with the Swin-B backbone outperforms SOTAs by 4.0% \mathcal{J}\&\mathcal{F} . Moreover, compared to the competitive baseline gro that combines GroundingDINO and SAM2 Ravi et al. 2024 for RVOS, ReferDINO demonstrates significant advantages in performance. For example, on Ref-Youtube-VOS dataset, ReferDINO with the Swin-T backbone improves the baseline by 12.0% \mathcal{J}\&\mathcal{F} . These results confirm the effectiveness of our designs over the foundation models. We summarize our contributions as follows: • To the best of our knowledge, ReferDINO is the first end-to-end approach for adapting foundational visual grounding models to RVOS. • We introduce three key innovations in ReferDINO to enable efficient object decoding, effective temporal understanding and accurate object segmentation. • Our ReferDINO significantly outperforms SOTA methods across five public RVOS benchmarks. 2 Related Works Figure 2 : Overall architecture of ReferDINO. Modules colored in blue are borrowed from GroundingDINO, while those in red are newly introduced in this work. Based on the frame-wise object features \{\mathcal{O}^{t}\}^{T}_{t=1} , our object-consistent temporal enhancer leverages the cross-modal text features to enable inter-frame object interaction. Then, our grounding-guided deformable mask decoder produces masks for the candidate objects, conditioned on location predictions, cross-modal text features and high-resolution feature maps. To further improve the efficiency in video processing, we introduce a confidence-aware query pruning strategy in the cross-modal decoder. Best view in color. Referring Video Object Segmentation. RVOS Gavrilyuk et al. 2018 ; Ding et al. 2023 ; Seo et al. 2020 aims to segment objects throughout the video based on text descriptions. MTTR Botach et al. 2022 firstly introduces the DETR paradigm Carion et al. 2020 into RVOS. Furthermore, ReferFormer Wu et al. 2022 proposes to produce the queries from the text description. On the top of this pipeline, follow-up works Han et al. 2023 ; Miao et al. 2023 ; Luo et al. 2023 ; Yuan et al. 2024 focus on modular improvements to improve cross-frame consistency and temporal understanding. Despite notable progress on specific datasets, these models are limited by insufficient vision-language understanding, and often struggle in unseen objects or scenarios. Recently, some works gro ; Huang et al. 2024 attempt to utilize GroundingDINO to identify objects in single frames and then apply independent segmentation models like SAM2 Ravi et al. 2024 to generate high-quality object masks. However, such a manner of model ensemble is inefficient and non-differentiable. Meanwhile, their performance almost completely relies on the static detection quality of GroundingDINO. In contrast, our ReferDINO is an end-to-end adaptation approach that benefits from both open-world knowledge of GroundingDINO and specific knowledge from RVOS data. Visual Grounding Foundation Model. The aim of visual grounding Hong et al. 2019 ; Hu et al. 2017 ; Liao et al. 2020 ; Sadhu et al. 2019 ; Wang et al. 2018 ; Wang et al. 2019 is to locate the most relevant object or region in an image, based on a natural language query. Recent foundational works aim to unify detection datasets and image-text datasets through region-text matching, and exploit large image-text data to increase the training vocabulary at scale. GLIP Li et al. 2022 formulates object detection as a grounding problem, utilizing additional grounding data to promote hierarchical vision-language alignment. GroundingDINO Liu et al. 2024 incorpoarates large-scale grounded pretraining

Read the complete text