METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships in Open-vocabulary Video Visual Relationship Detection

Yongqi Wang, Xinxiao Wu, Shuo Yang · 2025

Read METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships in Open-vocabulary Video Visual Relationship Detection by Yongqi Wang, Xinxiao Wu, Shuo Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06663.txt.

Introduction

Yongqi Wang, Xinxiao Wu, and Shuo Yang build a video system that names objects and their relations without a fixed list of allowed labels, in a 2025 paper. Older systems find objects first and label relations after, so early mistakes poison later steps. Their framework, METOR, lets the two tasks sharpen each other instead.

A module built on the CLIP vision-language model enriches text and object encodings with scene context, which helps with novel categories. An iterative loop then polishes object and relation readings in turns, using their mutual links. On two public video sets the method set new top scores. Open-vocabulary video reading matters for search and robotics, where the world will not stick to a label list.

Catalog record

CREATORYongqi Wang, Xinxiao Wu, Shuo Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06663

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships in Open-vocabulary Video Visual Relationship Detection Yongqi Wang, Xinxiao Wu, Shuo Yang Abstract Open-vocabulary video visual relationship detection aims to detect objects and their relationships in videos without being restricted by predefined object or relationship categories. Existing methods leverage the rich semantic knowledge of pre-trained vision-language models such as CLIP to identify novel categories. They typically adopt a cascaded pipeline to first detect objects and then classify relationships based on the detected objects, which may lead to error propagation and thus suboptimal performance. In this paper, we propose M utual E nhancemen T of O bjects and R elationships (METOR), a query-based unified framework to jointly model and mutually enhance object detection and relationship classification in open-vocabulary scenarios. Under this framework, we first design a CLIP-based contextual refinement encoding module that extracts visual contexts of objects and relationships to refine the encoding of text features and object queries, thus improving the generalization of encoding to novel categories. Then we propose an iterative enhancement module to alternatively enhance the representations of objects and relationships by fully exploiting their interdependence to improve recognition performance. Extensive experiments on two public datasets, VidVRD and VidOR, demonstrate that our framework achieves state-of-the-art performance. Codes are at https://github.com/wangyongqi558/METOR . Figure 1: (a) Existing methods adopt a cascaded pipeline. (b) Our method jointly models objects and relationships for their mutually enhancement. (c) Statistics of triplet errors on the VidVRD test set, categorized into object errors (OE) and relationship errors (RE). We also highlight the relationship errors caused by error propagation in the cascaded methods (OV-MMP and EOV-MMP), where the relationship is correctly classified using ground-truth object trajectories but misclassified using detected object trajectories. 1 Introduction Open-vocabulary video visual relationship detection (Open-VidVRD) Gao et al. 2023 focuses on detecting relationships between objects in videos, typically represented as triplets of the form \langle subject,relationship,object\rangle , following an open-vocabulary setting. In this setting, both object and relationship categories are divided into a base set and a novel set. The model is trained on the base set and is expected to generalize to the novel set during testing, enabling relationship detection in a wider range of open-vocabulary scenarios. Recent advances in pre-trained vision-language models Radford et al. 2021 ; Li et al. 2022 ; Li et al. 2023 have demonstrated significant potential in enhancing open-vocabulary tasks Liu et al. 2024 ; Wu et al. 2024b ; Fan et al. 2024 . By leveraging vast amounts of vision-language pairs during training, these models effectively encode rich semantic knowledge encompassing entities, actions, scenes, and relationships Fang et al. 2024 ; Wang et al. 2021 ; Liang et al. 2023 ; Gao et al. 2023 . In the Open-VidVRD task, these pre-trained models have been used to recognize novel object and relationship categories, thereby improving the generalization capability beyond predefined categories. Existing Open-VidVRD methods Gao et al. 2023 ; Yang et al. 2024 ; Wu et al. 2024a ; Wang et al. 2025 typically adopt a cascaded pipeline, first detecting objects and then classifying relationships based on the detected objects, as illustrated in Fig. 1 (a). This design often leads to error propagation, as the inaccuracy of object detection adversely affects the relationship classification, resulting in suboptimal performance. Fig. 1 (c) shows the number and causes of triplet errors for different methods on the VidVRD dataset. It is evident that the cascaded methods exhibit a significant increase in relationship errors due to error propagation, underscoring the inherent weaknesses of these methods. In this paper, we propose M utual E nhancemen T of O bjects and R elationships (METOR), a query-based unified framework for Open-VidVRD, which jointly models and mutually enhances object detection and relationship classification, as shown in Fig. 1 (b). It simplifies the Open-VidVRD process by adopting a unified modeling strategy, thereby mitigating the error propagation inherent in cascaded pipelines, and emphasizes the role of relationship context in enriching object representations, effectively exploiting the interdependence between objects and relationships to promote their mutual enhancement. As illustrated in Fig. 1 (c), our method reduces both object errors and relationship errors, highlighting the benefits of the proposed framework. To enhance the generalization ability to novel categories, we design a CLIP-based contextual refinement encoding module that captures the contextual information of objects and relationships to refine the encoding process. Specifically, we incorporate learnable object and relationship tokens into the CLIP visual encoder to capture their respective contexts. These contexts are then used to refine the CLIP-encoded text features and object queries, delivering instance-specific semantic knowledge to improve the adaptability in open-vocabulary scenarios. To fully exploit the interdependence between objects and relationships for recognition, we propose an iterative enhancement module to alternately enhance the representations of objects and relationships. Specifically, this module consists of multiple iterative enhancement layers, where each layer first uses object features to extract relationship features through spatio-temporal modeling and then uses the extracted relationship features to refine the object features, promoting a continuous mutual enhancement process that enables objects and relationships to iteratively improve each other’s representations. To summarize, the main contributions are as follows: • We propose METOR, a query-based unified framework that jointly models object detection and relationship classification to effectively exploit their interdependence to promote mutual enhancement, simplifying the process of Open-VidVRD. • We propose an iterative enhancement module that alternately enhances the representations of objects and relationships by using each other’s representations for more accurate recognition. • We design a CLIP-based contextual refinement encoding module that extracts contexts for objects and relationships to refine the encoding of text features and object queries for better open-vocabulary generalization. Figure 2: Overview of the proposed framework. 2 Related Work Video visual relationship detection (VidVRD) Shang et al. 2017 aims to detect relationships between objects over time in a given video, which has been widely applied to various visual understanding tasks Zhao et al. 2023 ; Nguyen et al. 2024 ; Rodin et al. 2024 . Numerous studies have explored various VidVRD methods, which can be broadly categorized into spatio-temporal modeling Qian et al. 2019 ; Tsai et al. 2019 ; Liu et al. 2020 ; Cong et al. 2021 , relationship refinement Shang et al. 2021 ; Chen et al. 2021 , video relationship debiasing Xu et al. 2022 ; Dong et al. 2022 ; Lin et al. 2024 , and end-to-end video relationship detection Zheng et al. 2022 ; Zhang et al. 2023 ; Jiang et al. 2024 . However, these methods are designed for close-set scenarios and struggle to generalize to open-vocabulary settings, thus being limited in pratical applications. Open-vocabulary VidVRD (Open-VidVRD) Gao et al. 2023 has emerged in recent years to extend VidVRD by testing novel categories, thereby enhancing its applicability to real-world scenarios. RePro Gao et al. 2023 , OV-MMP Yang et al. 2024 and UASAN Wu et al. 2024a design prompt learning or semantic alignment modules to better align visual and textual modalities. However, these methods rely on a close-set pre-trained trajectory detector for trajectory detection. Incorrect trajectories can lead to relationship classification errors, causing severe error propagation and hurting the overall performance. The method most related to ours is EOV-MMP Wang et al. 2025 ,

Read the complete text