ReLIC-SGG: Relation Lattice Completion for Open-Vocabulary Scene Graph Generation
Amir Hosseini, Sara Farahani, Xinyi Li, Suiyang Guang · 2026
Read ReLIC-SGG: Relation Lattice Completion for Open-Vocabulary Scene Graph Generation by Amir Hosseini, Sara Farahani, Xinyi Li, Suiyang Guang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22546.txt.
Introduction
Hosseini and three colleagues present ReLIC-SGG, describing scenes with open-ended relation phrases. Scene labels are always incomplete: many true relations go unmarked, and one interaction can be phrased several ways. Treating unmarked pairs as negatives teaches false lessons. The framework instead treats unmarked relations as unknowns to infer. A meaning lattice tracks which phrases are alike, imply others, or clash, guiding guesses from visual fit and graph context. Decoding guided by that lattice yields tight graphs recovering missing links and improving rare and unseen predicate recognition.
Catalog record
| CREATOR | Amir Hosseini, Sara Farahani, Xinyi Li, Suiyang Guang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42186 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22546
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReLIC-SGG: Relation Lattice Completion for Open-Vocabulary Scene Graph Generation Amir Hosseini, Sara Farahani, Xinyi Li, Suiyang Guang Abstract Open-vocabulary scene graph generation (SGG) aims to describe visual scenes with flexible relation phrases beyond a fixed predicate set. Existing methods usually treat annotated triplets as positives and all unannotated object-pair relations as negatives. However, scene graph annotations are inherently incomplete: many valid relations are missing, and the same interaction can be described at different granularities, e.g., on , standing on , resting on , and supported by . This issue becomes more severe in open-vocabulary SGG due to the much larger relation space. We propose ReLIC-SGG , a relation-incompleteness-aware framework that treats unannotated relations as latent variables rather than definite negatives. ReLIC-SGG builds a semantic relation lattice to model similarity, entailment, and contradiction among open-vocabulary predicates, and uses it to infer missing positive relations from visual-language compatibility, graph context, and semantic consistency. A positive-unlabeled graph learning objective further reduces false-negative supervision, while lattice-guided decoding produces compact and semantically consistent scene graphs. Experiments on conventional, open-vocabulary, and panoptic SGG benchmarks show that ReLIC-SGG improves rare and unseen predicate recognition and better recovers missing relations. 1 Introduction Scene graph generation (SGG) parses a visual scene into a structured graph, where nodes correspond to object instances and edges describe semantic relations between objects 17 ; 21 ; 29 ; 44 ; 41 . By explicitly modeling object-centric interactions, scene graphs provide an interpretable intermediate representation for visual question answering, visual reasoning, image retrieval, human-object interaction understanding, robotic manipulation, embodied navigation, and commonsense reasoning 40 ; 26 ; 3 . Over the past few years, SGG has evolved from closed-set predicate classification toward open-vocabulary relation understanding, where models are expected to recognize flexible and fine-grained relation phrases beyond a fixed predicate set 16 ; 33 ; 22 ; 13 ; 12 ; 4 ; 25 ; 32 ; 46 ; 15 . Despite this progress, most existing SGG methods still rely on a questionable supervision assumption: annotated triplets are treated as positives, while unannotated object-pair predicates are treated as negatives 44 ; 37 ; 36 ; 24 ; 45 ; 9 ; 10 ; 11 . This assumption is convenient for training but inconsistent with the nature of scene graph annotations. In real datasets, relation annotations are highly sparse and incomplete. For a given object pair, annotators usually label only one or a few salient relations, while many other valid relations are omitted 21 ; 17 ; 31 . For example, if an image contains a person standing on a skateboard , the annotation may include person-on-skateboard , but omit person-standing on-skateboard , person-supported by-skateboard , or skateboard-under-person . Similarly, for a cup on a table , the graph may annotate on but omit resting on , supported by , or above . Therefore, the absence of a relation label does not necessarily indicate that the relation is false. This issue is particularly harmful for open-vocabulary SGG. Compared with closed-set SGG, open-vocabulary SGG expands the predicate space from a small set of predefined labels to a much larger set of natural-language relation phrases 33 ; 22 ; 16 ; 13 ; 4 ; 25 ; 32 ; 46 . As a result, each object pair may correspond to multiple valid relation descriptions with different wording, granularity, and semantic scope. However, existing methods usually optimize relation prediction using standard classification or contrastive losses, which penalize unannotated candidate relations as negatives 36 ; 24 ; 45 . Consequently, a model may be discouraged from predicting valid but unannotated relations. This creates a fundamental conflict: open-vocabulary SGG aims to generate richer relation descriptions, but its supervision signal remains sparse, incomplete, and biased toward frequent predicates. Existing studies have made important progress in long-tail debiasing, semantic prototype learning, causal intervention, prompt learning, panoptic SGG, video SGG, 3D SGG, and embodied scene graph construction 36 ; 24 ; 31 ; 45 ; 42 ; 38 ; 43 ; 2 ; 1 ; 20 ; 35 ; 14 ; 18 . However, most of them still formulate SGG as a recognition problem over observed annotations. Even when vision-language models are introduced for open-vocabulary prediction, the learning objective often assumes that non-annotated relation phrases are incorrect 33 ; 22 ; 25 ; 46 ; 32 . This makes the model vulnerable to false-negative supervision. The problem is not only that rare predicates are under-represented, but that many valid predicates are never annotated as positives. In this work, we argue that the next step for open-vocabulary SGG is to move from closed-label relation classification to incomplete-label relation completion . Instead of asking only “which annotated predicate should be assigned to this object pair?”, we ask a different research question: how can an SGG model learn expressive open-vocabulary relations when many correct relations are missing from the annotations? This question shifts the focus from expanding the predicate vocabulary to learning under incomplete and ambiguous supervision. This view is connected to positive-unlabeled learning, partial-label learning, graph representation learning, and incomplete multimodal learning, where the absence of supervision should not be naively interpreted as negative evidence 8 ; 19 ; 6 ; 30 ; 7 . To address this problem, we propose ReLIC-SGG , a Rel ation- I ncompleteness-aware C ompletion framework for open-vocabulary SGG. The key idea is to treat unannotated relations not as definite negatives, but as latent variables whose labels should be inferred from visual evidence, language semantics, and graph context. Specifically, for each object pair, our model first proposes a set of open-vocabulary relation candidates using vision-language representations 33 ; 22 . It then constructs a semantic relation lattice that captures three types of predicate dependencies: semantic similarity, directional entailment, and contradiction. Based on this lattice, a latent relation completion module estimates the probability that an unannotated relation is a missing positive. Finally, a positive-unlabeled graph learning objective trains the model without incorrectly penalizing likely missing relations. The proposed framework is motivated by three observations. First, many relation phrases are semantically correlated rather than independent. For example, standing on entails on , while resting on and supported by are similar but not identical. Second, missing relations are not arbitrary; they can often be inferred from visual-language compatibility, neighboring graph context, and semantic consistency. Third, a useful scene graph should be semantically complete but not redundant: it should recover valid missing relations while avoiding contradictory or repetitive predictions. ReLIC-SGG is designed around these principles. Our contributions are summarized as follows: • We identify relation annotation incompleteness as a critical bottleneck for open-vocabulary SGG, where unannotated but valid relations are often incorrectly treated as negatives. • We propose ReLIC-SGG, a relation-incompleteness-aware framework that treats unannotated relations as latent variables and learns to complete missing relation labels. • We introduce a semantic relation lattice that models similarity, entailment, and contradiction among open-vocabulary predicates, enabling supervision transfer across relation phrases. • We design a positive-unlabeled graph learning objective and a lattice-guided graph decoding strategy to learn expressive yet compact scene graphs under incomplete supervision. • We introduce false-negative-aware evaluation protocols to measure whether a model can recover valid missing relations beyond standard recall-based metrics. 2 Related Work 2.1 Scene Graph Generation Scene graph generation aims to