Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation

Thong Thanh Nguyen, Xiaobao Wu, Yi Bin, Cong-Duy Nguyen, See-Kiong Ng, Anh Tuan Luu · 2024

Read Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation by Thong Thanh Nguyen, Xiaobao Wu, Yi Bin, Cong-Duy Nguyen, See-Kiong Ng, Anh Tuan Luu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07160.txt.

Introduction

To understand video, an AI must name the things in a scene and say how they relate over time, a task called temporal panoptic scene graph generation. Thong Thanh Nguyen, Xiaobao Wu, Yi Bin, Cong-Duy Nguyen, See-Kiong Ng, and Anh Tuan Luu argue that older methods waste a key clue: motion. They pool each tracked object over time and lose how it moved, though motion often gives the relation away.

Their method learns by contrast, pulling together object tracks that share the same actor-relation-target pattern. It pushes apart tracks that have been shuffled in time, and tracks from the same video with different patterns. Tests on video and 4D datasets showed clear gains over the best prior methods. The code is public.

Catalog record

CREATORThong Thanh Nguyen, Xiaobao Wu, Yi Bin, Cong-Duy Nguyen, See-Kiong Ng, Anh Tuan Luu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-07160

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation Thong Thanh Nguyen, Xiaobao Wu, Yi Bin, Cong-Duy Nguyen, See-Kiong Ng, Anh Tuan Luu Abstract To equip artificial intelligence with a comprehensive understanding towards a temporal world, video and 4D panoptic scene graph generation abstracts visual data into nodes to represent entities and edges to capture temporal relations. Existing methods encode entity masks tracked across temporal dimensions (mask tubes), then predict their relations with temporal pooling operation, which does not fully utilize the motion indicative of the entities’ relation. To overcome this limitation, we introduce a contrastive representation learning framework that focuses on motion pattern for temporal scene graph generation. Firstly, our framework encourages the model to learn close representations for mask tubes of similar subject-relation-object triplets. Secondly, we seek to push apart mask tubes from their temporally shuffled versions. Moreover, we also learn distant representations for mask tubes belonging to the same video but different triplets. Extensive experiments show that our motion-aware contrastive framework significantly improves state-of-the-art methods on both video and 4D datasets. Introduction The advent of autonomous agents, intelligent systems, and robots warrants a comprehensive understanding of real-world environments ( Ma et al. 2022 ; Driess et al. 2023 ; Raychaudhuri et al. 2023 ; Cheng et al. 2022 ; Li et al. 2023b ; Li et al. 2023a ) . Such understanding encompasses beyond merely recognizing individual entities, but also a sophisticated understanding of their relationships. To construct a detailed understanding, scene graph generation (SGG) research ( Li, Yang, and Xu 2022 ; Sudhakaran et al. 2023 ; Nag et al. 2023 ; Wang et al. 2024a ) has sought to provide relational perspective on scene understanding. In SGG frameworks, scene graphs utilize nodes to represent entities and edges to represent relationships, constructing a comprehensive and structured understanding of visual scenes. However, due to being primarily based on bounding boxes to denote entities, scene graphs fall short of replicating human visual perception with a lack of granularity ( Yang et al. 2023 ) . To overcome this limitation, panoptic scene graph generation ( Yang et al. 2022 ; Zhao et al. 2023 ) has been presented to expand the scope of SGG to incorporate pixel-level precise entity localization and thorough scene understanding including background components. Subsequently, because the temporal dimension undoubtedly provides richer information than the static spatial dimension, recent works ( Yang et al. 2023 ; Yang et al. 2024 ) have shifted attention to the domain of videos and 4D scenes, resulting in the tasks of panoptic video and 4D scene graph generation. Figure 1: State-of-the-art IPS+T - Convolution ( Yang et al. 2023 ) exhibits high R@50 scores for static relations, e.g. on , sitting on , and standing on , than dynamic relations, e.g. kicking , running on , and opening . In contrast, our method can perform effectively on both static and dynamic relations. Figure 2: Examples of temporal panoptic scene graph generation of state-of-the-art ( Yang et al. 2023 ; Yang et al. 2024 ) and our method. Popular methods ( Yang et al. 2023 ; Yang et al. 2024 ; Yang et al. 2022 ) for temporal panoptic scene graph generation produce entity masks tracked across the temporal dimension, i.e. mask tubes, then predict temporal relations among them. To conduct relation prediction, these methods encode the segmentation mask tubes, apply global pooling, then forward to a multi-layer perceptron for classifying their relations. However, such global pooling operation is well-known to be ineffective in representing temporal or motion patterns, which are useful for determining the relation among the entities. Consequently, this would result in higher misclassification rates of more dynamic relations ( Wang et al. 2023 ; Nag et al. 2023 ; Zhou et al. 2022 ) , as illustrated in Figure 1 . To encourage temporal representation learning, current research ( Nguyen et al. 2023b ; Liu et al. 2022 ; Zhou, Liu, and Wang 2023 ) uses contrastive learning for videos. However, they mainly seek to force two clips from the same video to be close together. As such, they mostly capture the semantics of visual scenes and disregard motions ( Chen et al. 2020 ) . Moreover, instead of paying attention to precise entity localization, they work upon frame-level representations. This would inadvertently inject motions from non-target entities into visual representations, which might not benefit relation classification in panoptic scene graph generation. In this paper, to encourage representation learning to capture motion patterns for temporal panoptic scene graph generation, we propose a novel contrastive learning framework that focuses on mask tubes of the segmented entities. First, we force a mask tube and the one of similar subject-relation-object but of a different video to obtain close representations. Since positive mask tubes originate from distinct video clips, the model cannot rely upon visual semantics to optimize the contrastive objective, but instead depends on the motion trajectory evolution, which is our target component for representation learning. Second, we propel negative mask tubes generated by temporally shuffling the original tubes. Moreover, we also push apart representations of mask tubes from the same video but belonging to different triplets. Because mask tubes of different triplets from a common video with close visual features are separated from each other, we once again motivate the model to generate representations that are less reliant upon visual semantics but motion-sensitive features. In addition, the visually similar negative mask tubes can play a role as hard negative samples, thus accelerating the contrastive learning process ( Chen, Zheng, and Song 2024 ) . Moreover, in order to implement our motion-aware contrastive learning framework, there is a need to quantify the relationship between mask tubes. This quantification marks a challenging problem as mask tubes are a sequence of segmentation masks that span over the sequence of video frames. Furthermore, mask tubes of two triplets might exhibit different lengths since two events often occur at different speed. Unfortunately, the popular pipeline of temporal pooling and then similarity estimation flattens the temporal dimension of the mask tubes and neglects their motion features. To resolve this problem, we consider mask tubes of two triplets as two distributions and seek the optimal transportation map between them, then utilize the transport distance as the distance between two triplets’ tubes. Such scheme of transporting can play a role of synchronizing the motion states of two triplets and takes advantage of the mask tubes’ evolutionary trajectory. To sum up, our contributions are as follows: • We propose a novel contrastive learning framework for temporal panoptic scene graph generation which pulls together entity mask tubes with similar motion patterns and pushes away those of distinct motion patterns. • We utilize optimal transport distance to estimate the relationship between two events’ mask tubes for the proposed contrastive framework. • Comprehensive experiments demonstrate that our framework outperforms state-of-the-art methods on both natural and 4D video datasets, especially on recognizing dynamic subject-object relations. Related Work Temporal panoptic scene graph generation. Traditional research mainly focuses on generating scene graphs for natural videos in which nodes represent objects and edges represent relations between objects. To localize object instances, the nodes are grounded by bounding boxes ( Wang et al. 2024a ; Nag et

Read the complete text