Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning
Minseok Kang, Minhyeok Lee, Minjung Kim, Jungho Lee, Donghyeong Kim, Sungmin Woo, Inseok Jeon, Sangyoun Lee · 2026
Read Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning by Minseok Kang, Minhyeok Lee, Minjung Kim, Jungho Lee, Donghyeong Kim, Sungmin Woo, Inseok Jeon, Sangyoun Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21559.txt.
Introduction
Kang and co-authors improve weakly-supervised video scene graph generation, which parses videos into relational triplets without box annotations. Off-the-shelf detectors propose every visible object, flooding relation models with pairs that never interact. Their fix learns a pair affinity score: how likely two objects are to interact. That score guides ranking at inference and modulates the context reasoning itself. A matching scheme uses vision-language grounding to clean up pseudo-labels. On the Action Genome benchmark, the additions lifted several baselines to state-of-the-art results.
Catalog record
| CREATOR | Minseok Kang, Minhyeok Lee, Minjung Kim, Jungho Lee, Donghyeong Kim, Sungmin Woo, Inseok Jeon, Sangyoun Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61106 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21559
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning Minseok Kang, Minhyeok Lee, Minjung Kim, Jungho Lee, Donghyeong Kim, Sungmin Woo, Inseok Jeon, Sangyoun Lee Abstract Weakly-supervised video scene graph generation (WS-VSGG) aims to parse video content into structured relational triplets without bounding box annotations and with only sparse temporal labeling, significantly reducing annotation costs. Without ground-truth bounding boxes, these methods rely on off-the-shelf detectors to generate object proposals, yet largely overlook a fundamental discrepancy from fully-supervised pipelines. Fully-supervised detectors implicitly filter out non-interactive objects, while off-the-shelf detectors indiscriminately detect all visible objects, overwhelming relation models with noisy pairs. We address this by introducing a learnable pair affinity that estimates the likelihood of interaction between subject–object pairs. Through Pair Affinity Learning and Scoring (PALS), pair affinity is incorporated into inference-time ranking and further integrated into contextual reasoning through Pair Affinity Modulation (PAM), enabling the model to suppress non-interactive pairs and focus on relationally meaningful ones. To provide cleaner supervision for pair affinity learning, we further propose Relation-Aware Matching (RAM), which leverages vision-language grounding to resolve class-level ambiguity in pseudo-label generation. Extensive experiments on Action Genome demonstrate that our approach consistently yields substantial improvements across different baselines and backbones, achieving state-of-the-art WS-VSGG performance. 1 Introduction Video scene graph generation (VSGG) aims to parse video content into structured triplets in the form of \langle subject, predicate, object \rangle , capturing spatial and temporal interactions between entities and supporting downstream tasks such as video understanding [ 6 , 22 ] , visual reasoning [ 18 , 5 , 7 ] , and video question answering [ 32 , 4 , 29 ] . Despite its significance, fully-supervised VSGG (FS-VSGG) methods face a critical scalability bottleneck: every frame requires exhaustive bounding box annotations for all objects together with their relationship triplets. To alleviate this burden, recent works have explored weakly-supervised VSGG (WS-VSGG), where supervision is provided in a reduced or indirect form. Representative approaches include PLA [ 1 ] , which leverages sparse temporal supervision with unlocalized triplets annotated only on the middle frame, where each triplet specifies the subject class, predicate, and object class without bounding boxes. Without ground-truth bounding boxes, WS-VSGG methods inevitably rely on off-the-shelf object detector [ 35 ] to generate object proposals, leading most approaches to adopt two-stage architectures [ 2 , 3 ] that first detect objects and then predict relations. However, existing WS-VSGG methods [ 1 , 33 ] largely adopt this pipeline without accounting for a fundamental shift in detection characteristics between the fully-supervised and weakly-supervised regimes. As illustrated in Figure 1 (a), a detector fine-tuned on scene graph datasets produces a curated set of proposals centered around relationally relevant objects. An off-the-shelf detector in WS-VSGG, by contrast, produces a far larger set of proposals dominated by objects irrelevant to any interaction. This shifts the burden of filtering non-interactive pairs from the detector to the relation prediction model. Figure 1 : Detection distribution and matching in FS-VSGG and WS-VSGG. (a) Per-frame detection statistics in the weakly annotated training set. Detections are categorized as True Match (spatial overlap with a ground-truth), False Match (class-matched without overlap), or Unmatched. (b) Detection and matching comparison for the triplet \langle person, holding, cup/glass/bottle \rangle . Fully-supervised detector produces relation-relevant proposals, whereas off-the-shelf detector generates many irrelevant objects, leading to false matches (e.g., cup 2) under class-level matching. Our method retains only the interaction-consistent instance (cup 1). In conventional two-stage WS-VSGG training pipelines, detector outputs are matched to triplet annotations to assign relation labels. Given unlocalized triplets \mathcal{G}^{u} for the middle frame t and detected object proposals \mathcal{D}_{t} , detections are matched to triplets based on class identity to construct pseudo-localized training pairs. As illustrated in Fig. 2(a), only matched pairs \mathcal{P}^{+} receive supervision while unmatched pairs \mathcal{P}^{-} are entirely discarded. While this convention is viable in FS-VSGG, it creates a severe train-test distribution gap in WS-VSGG. The relation prediction model is trained exclusively on interactive pairs, yet at inference it must operate over a detection space dominated by non-interactive pairs that were never observed during training. We address this by introducing a learnable pair affinity that captures the boundary between interactive and non-interactive pairs, whose scalar output measures whether a subject–object pair actively participates in an interaction regardless of the specific predicate. To jointly learn pair affinity alongside predicate classification, we introduce separate representations for the two objectives within the relation prediction model, each with distinct supervision signals. Unlike conventional training pipelines that discard unmatched pairs entirely (Figure 2 (a)), we retain them as negative supervision for pair affinity learning, together with matched pairs as positive (Figure 2 (b)). At inference, the learned pair affinity score is incorporated into triplet ranking to suppress non-interactive pairs. Figure 2 : Comparison of WS-VSGG training pipelines. \mathcal{G}^{u} : annotation of unlocalized triplets, \mathcal{D}_{t} : detected object proposals. (a) Only matched pairs \mathcal{P}^{+} are used for training; unmatched pairs \mathcal{P}^{-} are discarded. (b) RAM refines the matching, and both \mathcal{P}^{+} and \mathcal{P}^{-} are used to learn predicate classification and pair affinity jointly. While pair affinity score suppresses non-interactive pairs at the output level, the model’s internal contextual reasoning remains susceptible to their dominance. Modern VSGG architectures [ 2 , 3 , 9 , 19 ] perform contextual reasoning through spatial and temporal attention layers, where each pair refines its representation by attending to other pairs, yet non-interactive pairs equally participate and dilute relational context. To prevent non-interactive pairs from dominating this process, we propose Pair Affinity Modulation (PAM) , which gates attention between pairs based on their affinity, allowing interactive pairs to preferentially attend to each other. PAM is applied to both spatial and temporal attention layers, improving contextual reasoning within and across frames as a plug-and-play module compatible with existing attention-based architectures. Both pair affinity learning and modulation are supervised using the matched/unmatched partition. Since only unlocalized triplets are available, matching relies solely on class identity, which can generate incorrect pseudo labels when multiple instances of the same class coexist in a frame (Fig. 1(b)). These false matches corrupt both predicate classification and pair affinity supervision by introducing spurious positive labels for non-interacting pairs. We propose Relation-Aware Matching (RAM) to resolve this class-level matching ambiguity by leveraging vision-language (VL) models [ 14 , 16 ] to ground relational descriptions to specific object instances. RAM constructs relation-aware text queries from each triplet and uses cross-modal attention from the VL model to identify the most relation-consistent detection. As illustrated in Figure 1 (b), class-level matching indiscriminately assigns all same-class detections as positive, including non-interactive instances (cup 2), while RAM retains only the relation-consistent one (cup 1). RAM is a one-time preprocessing step that refines pseudo-labels for training, providing cleaner supervision for pair affinity learning with no overhead at inference. Together, our contributions form a unified framework that addresses the detection distribution shift in WS-VSGG. Extensive experiments on the Action Genome dataset demonstrate consistent improvements across multiple baselines and backbones, achieving state-of-the-art performance. 2 Related Work 2.1 Fully-Supervised Video Scene Graph Generation VSGG extends image-level scene graph generation to the temporal domain, producing scene graphs that capture dynamic relationships between objects across video frames. Existing methods operate at different granularities: video-level methods [ 24 ,