RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnesses
Minh Anh Nguyen, Quang Huy Tran, Bao Ngoc Le, Tuan Kiet Pham, Sui Yang Guang · 2026
Read RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnesses by Minh Anh Nguyen, Quang Huy Tran, Bao Ngoc Le, Tuan Kiet Pham, Sui Yang Guang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20823.txt.
Introduction
Minh Anh Nguyen and four colleagues present RelWitness, a design for open-vocabulary 3D scene graph generation under incomplete link supervision. The central difficulty is not word-list size but trust in labels, since many valid object-pair links go unannotated in 3D datasets. Their key concept is a link witness, a concrete visual-geometric cue that creates a link visible in the captured scene. Support needs contact and up-down order, containment needs enclosure, proximity needs metric closeness, and orientation needs facing use.
Catalog record
| CREATOR | Minh Anh Nguyen, Quang Huy Tran, Bao Ngoc Le, Tuan Kiet Pham, Sui Yang Guang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51004 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20823
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnesses Minh Anh Nguyen, Quang Huy Tran, Bao Ngoc Le, Tuan Kiet Pham, Sui Yang Guang Abstract Open-vocabulary 3D scene graph generation seeks to describe object instances and their relations with flexible natural-language predicates. The central difficulty is not only vocabulary expansion, but supervision reliability: relation annotations in 3D scene graph datasets are selective, and many valid object-pair relations are unannotated. Treating all unannotated relations as negatives suppresses useful missing relations, while completing labels from language plausibility or object co-occurrence can add physically unsupported edges. We propose RelWitness , a framework for open-vocabulary 3D scene graph generation from posed RGB-D sequences under incomplete relation supervision. The key concept is a relation witness : a concrete visual-geometric cue that makes a relation observable in the captured scene. Support relations require contact and vertical ordering; containment requires enclosure; proximity requires metric closeness; orientation requires facing direction; and stable relations should persist across views where both objects are visible. RelWitnessconstructs relation witness records from RGB views, depth maps, reconstructed 3D geometry, role-sensitive text, object-prior null views, and multi-view consistency. A visual-geometric witness verifier assigns unannotated relation candidates to verified missing positives, reliable negatives, or uncertain unlabeled cases. A witness-guided positive-unlabeled objective then learns from incomplete annotations without turning every missing label into a negative. We further introduce witness-consistent decoding and an RGB-D missing-relation audit protocol. Simulated manuscript-planning experiments on 3DSSG/3RScan and ScanNet-derived open-vocabulary splits show the intended behavior: improved unseen-relation recognition, higher witness precision, lower hallucination, and reduced redundant relation phrases. All numerical results are planning values and must be replaced by reproduced measurements before submission. 1 Introduction Scene graphs represent a visual environment as a structured graph whose nodes correspond to object instances and whose edges describe relations between objects. In 3D scenes, these graphs are particularly appealing because they connect semantic object identity with metric layout, support structure, containment, accessibility, and action-relevant context. A robot searching for a mug can benefit from relations such as mug on table , mug inside cabinet , or chair facing desk ; an embodied agent can use a scene graph to reason over object arrangements without processing every pixel or point; and an augmented-reality assistant can answer spatial queries using a compact object-relation representation 2 ; 54 ; 57 ; 16 . The field has recently moved from closed-vocabulary 3D scene graph prediction toward open-vocabulary 3D scene understanding. Language-aligned 3D features and 2D foundation models allow systems to query novel categories and relation phrases in reconstructed scenes 43 ; 50 ; 32 ; 16 ; 67 ; 27 . This shift is necessary: real indoor scenes contain relations that are fine-grained, compositional, and task-dependent. A table may be supporting a monitor, beside a chair, facing a sofa, and near a window. No fixed predicate list can cover all useful descriptions. However, open-vocabulary recognition exposes two coupled challenges that are largely hidden in conventional closed-set evaluation. First, relation plausibility gap : an open-vocabulary model can propose relations that sound correct for the object categories but are not physically present in the captured scene. A cup on table relation is common, but the RGB-D sequence may show the cup in a sink. A chair under table edge is plausible indoors, but the chair may actually be beside the table. Second, incomplete observability supervision : an unannotated relation is not a single type of training example. It may be a true missing positive, a true negative, or a relation whose truth is not observable from the available views. A scan may annotate monitor on desk but omit monitor supported by desk ; it may annotate chair near table but omit chair facing table . Penalizing all missing labels suppresses useful relations, while accepting all plausible phrases creates hallucinated graph edges. These two challenges suggest that vocabulary expansion alone is insufficient. A useful open-vocabulary 3D SGG method must answer a more precise question: When an unannotated relation phrase is plausible, what visual-geometric cue in the captured RGB-D scene makes it safe to learn from? We answer this question with relation witnesses . A relation witness is a physically interpretable cue that makes a relation observable. It is not simply a high classifier score, attention map, or text-image similarity. It is a relation-family-specific check: contact for on , enclosure for inside , vertical order for under , metric closeness for near , orientation for facing , boundary contact for attached to , and multi-view persistence for stable spatial relations. Some phrases, such as ownership or intended use, may not have an observable witness in a static RGB-D scan and should remain uncertain rather than being forced into positives or negatives. This witness perspective changes the story of open-vocabulary 3D SGG. Existing systems are increasingly good at proposing relation phrases, but proposing is not the same as supervising. RelWitnessfocuses on the missing-supervision step: among many semantically plausible relation candidates, which ones are physically observable enough to become training signals? This distinction separates RelWitnessfrom open-set querying methods such as Open3DSG 32 , object-centric mapping systems such as ConceptGraphs 16 , functional graph methods such as OpenFunGraph 67 , and online graph construction methods such as FROSS 27 . Those works expand what can be queried or built; RelWitnessasks how incomplete relation labels should be corrected without hallucinating unsupported edges. Given a posed RGB-D sequence, RelWitnessfuses object instances into a global 3D scene, proposes open-vocabulary relation candidates, parses each relation into witness families, and constructs a relation witness record. The record includes selected RGB views, depth-based cues, 3D geometric probes, role consistency, object-prior null scores, multi-view stability, and rendered witness traces. A momentum witness verifier uses these records to maintain three dynamic sets: verified missing positives, reliable negatives, and uncertain candidates. The final model is trained with a witness-guided positive-unlabeled objective and decoded with witness-consistent redundancy suppression. Our contributions are: • We formulate open-vocabulary 3D SGG under incomplete supervision as a problem of physical observability , introducing relation witnesses as the unit for deciding whether unannotated relations can become supervision. • We design a visual-geometric witness verifier that combines RGB views, depth cues, reconstructed 3D geometry, role-sensitive relation text, object-prior null tests, and multi-view persistence. • We propose witness-guided positive-unlabeled learning with a family-balanced witness memory that stores verified missing positives, reliable negatives, uncertain candidates, and their witness traces. • We introduce witness-consistent graph decoding and an RGB-D missing-relation audit protocol that measures witness precision, hallucination, redundancy, and multi-view agreement. • We provide a complete experimental design with simulated planning results showing how the method should be evaluated on 3DSSG/3RScan and ScanNet-derived open-vocabulary benchmarks. 2 Related Work 3D Scene Graph Generation. 3D scene graphs were introduced as structured representations that connect semantic objects, spatial layout, and camera geometry 2 . They have since been used for metric-semantic mapping, localization, robotics, navigation, and planning 48 ; 16 . Learning-based 3D SGG methods predict object relations from point clouds or RGB-D reconstructions, with ScanNet and 3DSSG/3RScan providing influential indoor benchmarks 8 ; 54 . SceneGraphFusion incrementally predicts and fuses relations from RGB-D sequences 57 . The geometric backbone of this literature also draws on point-cloud encoders and transformers 44 ; 45 ; 53 ; 68 .