Click2Graph: Interactive Panoptic Video Scene Graphs from a Single Click
Raphael Ruschel*, Hardikkumar Prajapati*, Md Awsafur Rahman, B. S. Manjunath · 2025
Read Click2Graph: Interactive Panoptic Video Scene Graphs from a Single Click by Raphael Ruschel*, Hardikkumar Prajapati*, Md Awsafur Rahman, B. S. Manjunath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15948.txt.
Introduction
Raphael Ruschel, Hardikkumar Prajapati, Awsafur Rahman, and B. S. Manjunath introduce Click2Graph, an interactive system that builds video scene graphs from a single user cue. A scene graph lists objects and their relations, such as a person holding a cup, over time. Existing generators run as closed pipelines with no human guidance, while promptable segmentation models accept clicks but do no relational reasoning.
From one click or bounding box, Click2Graph segments and tracks the subject, discovers interacting objects on its own, and predicts subject-object-relation triplets that stay consistent over time. A discovery module proposes objects conditioned on the subject, and a classification head reasons jointly about entities and predicates. Tests on a standard benchmark establish a strong user-guided foundation.
Catalog record
| CREATOR | Raphael Ruschel*, Hardikkumar Prajapati*, Md Awsafur Rahman, B. S. Manjunath |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-15948
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Click2Graph: Interactive Panoptic Video Scene Graphs from a Single Click Raphael Ruschel*, Hardikkumar Prajapati*, Md Awsafur Rahman, B. S. Manjunath Abstract State-of-the-art Video Scene Graph Generation (VSGG) systems provide structured visual understanding but operate as closed, feed-forward pipelines with no ability to incorporate human guidance. In contrast, promptable segmentation models such as SAM2 enable precise user interaction but lack semantic or relational reasoning. We introduce Click2Graph , the first interactive framework for Panoptic Video Scene Graph Generation (PVSG) that unifies visual prompting with spatial, temporal, and semantic understanding. From a single user cue, such as a click or bounding box, Click2Graph segments and tracks the subject across time, autonomously discovers interacting objects, and predicts ⟨ s u b j e c t , o b j e c t , p r e d i c a t e ⟩ \langle subject,object,predicate\rangle triplets to form a temporally consistent scene graph. Our framework introduces two key components: a Dynamic Interaction Discovery Module that generates subject-conditioned object prompts, and a Semantic Classification Head that performs joint entity and predicate reasoning. Experiments on the OpenPVSG benchmark demonstrate that Click2Graph establishes a strong foundation for user-guided PVSG, showing how human prompting can be combined with panoptic grounding and relational inference to enable controllable and interpretable video scene understanding. 1 Introduction Understanding not only what appears in a video but how entities interact is a core challenge in intelligent perception. This capability is desired in applications in robotics, autonomous agents, assistive systems, and surveillance, where downstream decisions depend on correctly interpreting actions, intentions, and relationships. Scene Graph Generation (SGG) has emerged as a powerful representation for such structured understanding, evolving from static image reasoning Xu et al. 2017 ; Zellers et al. 2018 to dynamic, video-based formulations that capture temporal context Cong et al. 2021 ; Khandelwal and Sigal 2022 . More recently, panoptic scene graph generation has advanced grounding fidelity by replacing bounding boxes with pixel-level masks Yang et al. 2022 ; Yang et al. 2023b , enabling fine-grained grounding of classes, especially for objects with irregular shapes (commonly refered as ”stuff” classes), such as floor and sky. Figure 1 : On the left example, the user clicked on the \langle\text{{dog}}\rangle , and Click2Graph segmented the \langle\text{{carpet}}\rangle and predicted the \langle\text{{sitting}}\rangle activity. On the right, we have a prompt on \langle\text{{child}}\rangle which yields \langle\text{{dog}}\rangle , \langle\text{{playing}}\rangle as associated object and activity. Despite these advances, existing video SGG and PVSG pipelines remain fully automated and closed-loop. Once a model overlooks an occluded object, misclassifies a rare interaction, or drifts during tracking, the user has no mechanism to intervene. This lack of controllability is problematic in complex or safety-critical environments, where correcting errors or directing model attention is essential. At the same time, a new class of promptable segmentation models, most notably SAM and SAM2 Kirillov et al. 2023 ; Ravi et al. 2024a , has demonstrated the power of direct visual prompting . With a simple click or box, users can obtain precise, temporally consistent segmentation masks. Yet these models are inherently class-agnostic and relation-agnostic: they determine where objects are but not what they are or how they interact. This disconnect reveals a fundamental gap: current PVSG systems lack user guidance, and current interactive segmentation models lack semantic structure. We address this gap with Click2Graph , the first framework for user-guided Panoptic Video Scene Graph Generation . From a single visual cue, such as clicking a subject in any frame, Click2Graph: 1. Segments and tracks the prompted subject across time, 2. Autonomously discovers and segments interacting objects, and 3. Predicts \langle subject,object,predicate\rangle relationships to form a temporally consistent scene graph. Figure 1 illustrates how distinct scene graphs can be produced depending on which entity the user prompts, highlighting the controllability of the system. Click2Graph introduces two components that supply the missing semantic and relational reasoning. A Dynamic Interaction Discovery Module (DIDM) generates subject-conditioned object prompts, enabling automatic discovery of entities participating in interactions. A Semantic Classification Head (SCH) performs joint subject, object, and predicate inference over the discovered segments, producing structured scene graph outputs. Together, these components elevate promptable segmentation from geometric mask extraction to full panoptic video scene graph generation. Our contributions are summarized as follows: • User-Guided Panoptic Video Scene Graphs. We introduce the first interactive PVSG framework that converts a single visual prompt into a temporally consistent panoptic scene graph, enabling controllable, interpretable video analysis. • Dynamic Interaction Discovery. We propose a novel module that generates subject-conditioned prompts to discover interacting objects, naturally supporting multi-subject and multi-object reasoning. • Semantic Reasoning atop Promptable Segmentation. A dedicated classification head predicts subject–object pairs labels and the relationship between them, bridging the gap between prompt-based segmentation and structured semantic inference. Click2Graph establishes a new paradigm for video scene understanding by combining human guidance, pixel-level grounding, and relational reasoning within a unified architecture. As shown in our experiments on the OpenPVSG benchmark, this paradigm enables controllable and interpretable scene graph generation while offering a practical path toward real-world interactive video analytics. 2 Related Works Table 1 : A comparative analysis of Scene Graph Generation paradigms. Click2Graph is the first to unify video-level temporal reasoning, panoptic-level spatial precision, and user-guided visual prompting for end-to-end tracking and relationship prediction. Method Modality Granularity Interaction Type End-to-End Tracking Relationship Prediction Traditional SGG (e.g., MOTIFS Zellers et al. 2018 ) Image Box None N/A Yes Video SGG (e.g., STTran Cong et al. 2021 ) Video Box None Yes Yes Panoptic SGG (e.g., PSGFormer Yang et al. 2022 ) Image Mask None N/A Yes Panoptic Video SGG (PVSG) Yang et al. 2023b Video Mask None Yes Yes Text-Prompted SGG (e.g., VLPrompt Zhou et al. 2023 ) Image Mask Text N/A Yes Click2Graph (Ours) Video+Image Mask Visual (Click/Box) Yes Yes Our work lies at the intersection of video scene graph generation, panoptic-level scene understanding, and interactive visual analysis. Below, we position Click2Graph within each of these domains. Table 1 shows a summary of the related domains. 2.1 Advances in Video Scene Graph Generation Scene Graph Generation (SGG) was first developed for static images Xu et al. 2017 ; Zellers et al. 2018 , and later extended to videos (VidSGG) to capture temporal dynamics Khandelwal and Sigal 2022 ; Wang et al. 2022 ; Li et al. 2022b ; Lu et al. 2023 ; Lin et al. 2024 ; Feng et al. 2023 ; Wang et al. 2024 ; Zhang et al. 2023 . Transformer-based approaches such as STTran Cong et al. 2021 , DDS Iftekhar et al. 2025 , and VSG-Net Ulutan et al. 2020 improved long-range temporal reasoning and robustness to clutter. Another thread of work addresses the heavy long-tail distribution of predicates through debiasing methods such as TEMPURA Nag et al. 2023 and VISA Li et al. 2025b , while DiffVSGG Chen et al. 2025 frames video SGG as an iterative denoising problem. Although these methods advance automated scene graph reasoning, they operate as closed-loop systems: once the model misdetects or misclassifies an entity, the user cannot intervene. Click2Graph introduces this missing interactive dimension, enabling subject-specific, user-directed scene graph construction. 2.2 Panoptic-Level