Not All Relations Rotate Alike: Transformation-Aware Decoupling for Viewpoint-Robust 3D Scene Graph Generation

Jingjun Sun, Chaowei Wang, Zhirui Liu, Jiaxu Tian, Ming Yang, Yaoxing Wang, Shan Gao · 2026

Read Not All Relations Rotate Alike: Transformation-Aware Decoupling for Viewpoint-Robust 3D Scene Graph Generation by Jingjun Sun, Chaowei Wang, Zhirui Liu, Jiaxu Tian, Ming Yang, Yaoxing Wang, Shan Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27412.txt.

Introduction

Sun leads a team fixing a subtle failure in three-dimensional scene graph generation: relations should transform correctly when the viewer moves. A scene graph lists objects and the relations between them. Under yaw rotations of the viewpoint, directional relations like left or behind must rotate with the frame, while contact and support relations like standing on should stay fixed. Current models entangle the two behaviours and mispredict after rotation. Their Transformation-Aware Decoupling framework splits relation reasoning into stable and directional branches in this computer vision paper of 2026.

Stable object features support the split, and transformation-specific descriptors plus group-aware training signals keep the branches distinct. Merged, they produce the standard multi-label relation predictions. On the standard benchmark the method reaches state-of-the-art robustness under yaw changes without any rotation augmentation during training. Relations learn their own transformation rules.

Catalog record

CREATORJingjun Sun, Chaowei Wang, Zhirui Liu, Jiaxu Tian, Ming Yang, Yaoxing Wang, Shan Gao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27412

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Not All Relations Rotate Alike: Transformation-Aware Decoupling for Viewpoint-Robust 3D Scene Graph Generation Jingjun Sun, Chaowei Wang, Zhirui Liu, Jiaxu Tian, Ming Yang, Yaoxing Wang, Shan Gao Abstract 3D Scene Graph Generation (3DSGG) represents 3D scenes as structured object-relation-object graphs, providing a compact relational abstraction for spatial understanding. In embodied intelligence settings, the same 3D scene may be observed by agents from viewpoints that differ by yaw rotations. However, current 3DSGG models often fail to produce relation predictions that follow the expected transformation behavior under such viewpoint shifts. This behavior reveals an empirical mismatch related to predicate-level transformation heterogeneity: directional predicates such as left , front , right , and behind should transform with the observation frame, whereas most contact, support, and semantic predicates such as standing on and attached to should remain stable. To reduce this mismatch, we propose Transformation-Aware Decoupling (TAD), a viewpoint-robust 3DSGG framework that decouples relation reasoning according to predicate transformation behavior and is supported by viewpoint-stable object representations. TAD decomposes relation reasoning into two parts: one learns cues that should stay stable across viewpoints, while the other learns directional cues that should change with the observation frame. The two parts are merged for standard multi-label predicate prediction. Transformation-specific descriptors and group-aware auxiliary supervision encourage the two branches to capture complementary relation cues. Extensive experiments on 3DSSG show that TAD achieves state-of-the-art robustness under yaw viewpoint changes without training-time rotation augmentation, while maintaining competitive performance under the standard benchmark. The project page is available at https://tad-predicate.github.io/ . Figure 1 : Transformation-Aware Decoupling (TAD) is a predicate-level framework for viewpoint-robust 3D scene graph generation. Under yaw viewpoint changes, directional relations such as left should transform, while yaw-invariant relations such as standing on remain stable. TAD decouples them into separate invariant-relation and directional-relation branches, resolving the conflict of entangled learning. 1 Introduction Understanding a 3D scene requires not only recognizing individual objects, but also capturing the relational structure among them [ 1 , 27 ] . 3D Scene Graph Generation (3DSGG) addresses this challenge by representing a scene as a graph: objects are modeled as nodes, and their spatial relationships are modeled as edges. By transforming raw 3D observations into graph-based scene representations, 3DSGG serves as a fundamental step toward spatial reasoning and embodied intelligence [ 19 , 10 , 12 , 15 , 2 ] , providing high-level scene abstractions for downstream applications such as robotic navigation and augmented reality [ 19 , 22 ] . Driven by the demands of these downstream applications, a rich body of 3DSGG frameworks has been developed. Most 3DSGG methods infer object relations from pairwise object features and relative geometry, increasingly enhanced by visual-semantic priors and instance-level recognition [ 27 , 32 , 39 , 30 , 10 , 13 , 11 , 40 ] . Evaluated under a standard, canonical scene orientation, these methods have demonstrated remarkable performance in parsing complex 3D environments. However, in real-world embodied scenarios, the same scene is naturally observed from multiple viewpoints, with yaw rotations about the gravity-aligned vertical axis being the dominant source of variation. Existing frameworks largely overlook this multi-view reality, implicitly assuming a static observation frame without explicitly accounting for viewpoint changes in relation prediction. This raises a critical question: how robust are existing 3DSGG frameworks to yaw-induced viewpoint changes? Through controlled experiments under a range of yaw-induced viewpoint changes, we reveal that existing frameworks remain relatively robust to opposite-viewpoint changes, where directional predicates undergo simple reversals, but suffer substantial degradation beyond this simple reversal case. These results point to a mismatch between the transformation behavior of 3D predicates and the way relation representations are conventionally modeled. As illustrated in Fig. 1 , 3D predicates respond differently to yaw-induced viewpoint changes. Relations such as standing on should remain semantically stable, whereas directional relations such as left , right , front , and behind should change predictably with the observation frame. Conventional relation models, however, typically embed these predicates in a shared relation space without distinguishing their transformation behaviors. This entangles cues that should be preserved with cues that should change predictably, reducing the robustness of relation prediction under viewpoint changes. Building on these observations, we propose TAD ( Transformation-Aware Decoupling ), a 3DSGG framework that decouples relation learning according to the transformation behaviors of 3D predicates. TAD separates the learning of relation cues according to predicate-specific transformation behaviors, using transformation-specific descriptors to guide complementary relation representations. It models viewpoint-stable cues that should be preserved across viewpoint changes separately from direction-sensitive cues that should vary predictably with the observation frame. The relation representations produced by these decoupled pathways are then fused for standard multi-label predicate prediction. To support this decoupled relation learning, TAD further incorporates a Viewpoint-Stable Object Encoder (VSOE) , which extracts rotation-stable object features and contextualizes them into scene-aware node representations for downstream relation reasoning. By decoupling relation learning according to predicate-specific transformation behaviors, TAD better accommodates viewpoint-induced predicate changes and yields more robust scene graph predictions. Our core contributions are threefold: 1) We identify heterogeneous predicate transformation behaviors as a key obstacle to robust 3D Scene Graph Generation under yaw-induced viewpoint changes, motivating transformation-aware relation modeling. 2) We propose TAD, a transformation-aware 3DSGG framework. Supported by viewpoint-stable object representations, TAD employs parameter-decoupled relation branches with orthogonal feature regularization and group-aware auxiliary supervision to disentangle viewpoint-stable and direction-sensitive relation cues for transformation-aware reasoning. 3) Extensive quantitative and qualitative experiments demonstrate that our method achieves state-of-the-art relation robustness under yaw-induced viewpoint changes, while remaining on par with existing SOTA methods on the standard 3DSGG benchmark. 2 Related Work 2.1 3D Scene Graph Generation 3D scene graph generation extends 2D SGG [ 38 , 34 , 23 ] to 3D point clouds, aiming to construct structured object-relation-object graphs from 3D scene geometry. Early 3DSGG methods established benchmark datasets and GNN-based prediction frameworks with instance-level point features and pairwise geometric descriptors [ 27 , 26 , 32 , 39 , 40 ] . Recent methods have advanced 3DSGG from different perspectives. VL-SAT [ 30 ] introduced visual-linguistic semantics assisted training, where a multi-modal oracle transfers visual and linguistic knowledge to a 3D model during training while inference uses only 3D inputs. OCRL [ 11 ] emphasized discriminative object representation learning via object-centric contrastive pretraining, showing that stronger object features benefit both object classification and relationship prediction. Other recent efforts improve RGB-based 3DSGG with segmentation-guided feature aggregation and statistical confidence rescoring [ 36 ] , or explore scene-graph-level alignment for downstream registration [ 20 , 33 , 4 ] . Open-vocabulary and VLM/LLM-based 3D scene graph approaches, alongside dynamic 3D Gaussian Splatting representations [ 29 ] , further study generalization beyond closed predicate sets and view-invariant graph construction [ 13 , 10 , 5 , 16 ] . Despite these advances, existing 3DSGG methods largely overlook predicate-level transformation heterogeneity under yaw viewpoint changes, and typically predict all predicates in a shared benchmark label space without distinguishing relations that should remain stable from those that should transform. We identify this heterogeneity as a key factor for viewpoint-robust 3DSGG and address it with a transformation-aware relation modeling framework that decouples yaw-invariant and direction-sensitive relation cues while preserving unified multi-label predicate prediction. 2.2 Rotation-Robust Relation Modeling Relation modeling

Read the complete text