TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring

Zhu Xu, Ting Lei, Zhimin Li, Guan Wang, Qingchao Chen, Yuxin Peng, Yang Liu · 2025

Read TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring by Zhu Xu, Ting Lei, Zhimin Li, Guan Wang, Qingchao Chen, Yuxin Peng, Yang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04943.txt.

Introduction

Zhu Xu and co-authors teach machines to diagram moving scenes without paying for frame-by-frame labels, in this 2025 paper. Dynamic scene graphs list each video frame's objects and their relations, a rich description for search and robotics. Full labeling is ruinously expensive, so weak supervision trains from one frame's unlocalized graph per video. Standard pipelines lean on a ready-made object detector for first guesses, but detectors trained on still photos misfire amid motion and relations.

Their method transfers relation-aware knowledge across time to shore up the weak spots. Better pseudo-labels early mean better graphs at the end. The aim is video understanding that scales to real footage volumes, where hand annotation will never reach.

Catalog record

CREATORZhu Xu, Ting Lei, Zhimin Li, Guan Wang, Qingchao Chen, Yuxin Peng, Yang Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04943

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring Zhu Xu, Ting Lei, Zhimin Li, Guan Wang, Qingchao Chen, Yuxin Peng, Yang Liu Abstract Dynamic Scene Graph Generation (DSGG) aims to create a scene graph for each video frame by detecting objects and predicting their relationships. Weakly Supervised DSGG (WS-DSGG) reduces annotation workload by using an unlocalized scene graph from a single frame per video for training. Existing WS-DSGG methods depend on an off-the-shelf external object detector to generate pseudo labels for subsequent DSGG training. However, detectors trained on static, object-centric images struggle in dynamic, relation-aware scenarios required for DSGG, leading to inaccurate localization and low-confidence proposals. To address the challenges posed by external object detectors in WS-DSGG, we propose a Temporal-enhanced Relation-aware Knowledge Transferring (TRKT) method, which leverages knowledge to enhance detection in relation-aware dynamic scenarios. TRKT is built on two key components:(1) Relation-aware knowledge mining : we first employ object and relation class decoders that generate category-specific attention maps to highlight both object regions and interactive areas. Then we propose an Inter-frame Attention Augmentation strategy that exploits optical flow for neighboring frames to enhance the attention maps, making them motion-aware and robust to motion blur. This step yields relation- and motion-aware knowledge mining for WS-DSGG. (2) we introduce a Dual-stream Fusion Module that integrates category-specific attention maps into external detections to refine object localization and boost confidence scores for object proposals. Extensive experiments demonstrate that TRKT achieves state-of-the-art performance on Action Genome dataset. Our code is avaliable at https://github.com/XZPKU/TRKT.git . 1 Introduction Dynamic Scene Graph Generation (DSGG) aims to represent complex visual scenes in a video sequence as structured graphs, with nodes representing object instances while edges capture the relationships between objects, which is valuable for visual-language tasks like Human Object Interaction Detection [ 15 , 13 , 14 ] , Visual Grounding [ 37 , 20 ] , Visual Question Answering [ 26 , 5 ] . However, annotating for video scene graphs is highly challenging and resource-intensive, hindering the scaling of DSGG. To address it, weakly supervised DSGG [ 3 ] (WS-DSGG) has been proposed, which relies on only one unlocalized scene graph from one frame of video as supervision, making it a practical solution for expanding DSGG to more complex video data. Figure 1 : Performance Comparison between different methods in object detection and WS-DSGG. Figure 2 : Comparison between (a) existing off-the-shelf object detection results and (b) improved object detection results with visual cues under weak supervision. Existing WS-DSGG approaches, such as the state-of-the-art model PLA [ 3 ] , rely on off-the-shelf external object detectors to generate object labels, which are then used to construct pseudo-localized scene graphs for training a fully supervised DSGG model. However, these external detectors often struggle with the DSGG task due to two key limitations: (1) video frames contain dynamic motion and potential blurring, whereas external detectors are trained on static, object-centric images, making them ill-suited for such scenarios; (2) external detectors are trained solely on object annotations, overlooking crucial relational cues necessary for dynamic scene graph data, leading to biased detection bounding boxes that miss interactive object boundary areas. These limitations lead to inaccurate object localization and missing detections, ultimately degrading the quality of pseudo labels and hindering final DSGG performance. To systematically assess the impact of object detection and predicate prediction on DSGG performance, we conduct an evaluation using PLA. As shown in Fig. 1 , combining PLA’s object detection results with oracle relation labels leads to a modest 0.9% improvement in DSGG performance. In stark contrast, substituting PLA’s object detections with oracle detections—thus providing accurate object pairs for training—yields a substantial 61.9% performance boost. These findings highlight the critical role of external detector quality in WS-DSGG performance. To address the challenges mentioned above, we propose an approach termed Temporal-enhanced Relation-aware Knowledge Transferring (TRKT) for the WS-DSGG task. Our method strategically targets the identified external detection weaknesses by mining and integrating knowledge, with the aim to make the detection results both relation-aware and motion-aware. Specifically, we employ a class-sensitive object and relation decoder the extract class-sensitive attention maps for potential objects, using relation-aware weak supervision tailored to the specific context of the WS-DSGG task. We introduce both object and relation tokens in the decoder to explicitly encapsulate objects as well as the relations between objects within the scene, enhancing the model’s ability to capture and prioritize the relation regions of corresponding objects, thus improving the overall quality of the scene graph. Furthermore, we utilize optical flow information to provide a temporal cue on object movement across video frames, which is crucial for maintaining coherence in object tracking. We then augment the class-sensitive attention maps between adjacent frames with these temporal cues, alleviating the problems of blurring and occlusion that arise in dynamic scenes. To this end, we generate class-sensitive attention maps that are both relation-aware and motion-aware. However, effectively transferring the knowledge embedded in these attention maps to enhance external detection results presents a significant challenge. To address this, we propose a Dual-stream Fusion Module (DFM), which consists of two key components: (1) Confidence Boosting Module (CBM): This module mitigates the issue of low-confidence detections by leveraging class-sensitive attention maps to reassess and refine the confidence scores of external detections. For each high-confidence category identified by the class decoder, CBM validates the corresponding detection scores, reducing the risk of missed detections caused by low-confidence predictions. As illustrated in Fig. 2 , CBM enhances the confidence score of the detected “laptop” in the right image. (2) Localization Refinement Module (LRM): This module integrates temporal and relational information from class-sensitive attention maps to refine bounding box coordinates, addressing localization inaccuracies. This process employs Weighted Box Fusion [ 22 ] to enhance object localization. As shown in Fig. 2 , LRM corrects an undersized “person” bounding box by incorporating semantic attention cues. In summary, the main contributions are as follows: (1) We demonstrate the significant impact of object detection quality on the performance of WS-DSGG, highlighting the key limitations of existing methods. (2) We propose the novel approach Temporal-enhanced Relation-aware Knowledge Transferring (TRKT), which utilizes class-sensitive knowledge that are both relation-aware and motion-aware, to enhance the object detection specifically for WS-DSGG. (3) We further introduce the Dual-stream Fusion Module (DFM) to incorporate the above attention maps to improve the external detection results, which comprises Localization Refinement Module (LRM) to improve bounding box accuracy and the Confidence Boosting Module (CBM) to dynamically adjust confidence scores. (4) We validate the effectiveness of our TRKT framework through extensive experiments, underscoring the critical role of improved object detection accuracy in enhancing WS-DSGG performance. 2 Related Work 2.1 Dynamic Scene Graph Generation DSGG is a spatial-temporal scene understanding task that aims to leverage temporal context to analyze the objects along with their dynamic relationships [ 10 ] . Two distinct task formulations exist for DSGG based on their output format: (1) Traclet-level [ 7 , 8 , 9 ] : each node of output scene graph represents one object traclet in the input video. (2) Frame-level [ 17 , 6 , 4 , 24 , 1 ] : Each node of scene graph represents one object bounding box, and each video

Read the complete text