RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots

Giorgia Modi, Davide Buoso, Giuseppe Averta, Daniele De Martini · 2026

Read RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots by Giorgia Modi, Davide Buoso, Giuseppe Averta, Daniele De Martini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18197.txt.

Introduction

Giorgia Modi's 2026 paper builds 3D scene graphs for indoor robots from RGB images alone, with no depth sensor. Perception and planning share one structured view holding object meanings, 3D geometry, relations, and multi-view evidence. On the Replica set the RGB-only pipeline matches depth-based baselines on F1 score. Active viewpoint choice on ReplicaCAD spots over double the objects of a geometric frontier baseline under the same budget. Added outside views bootstrap the graph at no extra exploration cost.

Catalog record

CREATORGiorgia Modi, Davide Buoso, Giuseppe Averta, Daniele De Martini
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20559
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18197

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots Thanks: *Correspondent author: [email protected] Thanks: This work was supported by the EPSRC Impact Acceleration Account (IAA), and the UKRI InnovateUK project ‘A general-purpose configuration management system for highly configurable systems validated in automated logistics’. Giorgia Modi, Davide Buoso, Giuseppe Averta, Daniele De Martini Abstract Current approaches to 3D scene graph generation rely on dedicated depth sensors, such as LiDAR or RGB-D cameras, for metric 3D reconstruction. This limits deployment to specialized robotic platforms and excludes settings where only RGB cameras are available, such as fixed external infrastructure. Existing pipelines also typically operate on passively collected observation trajectories, rather than selecting viewpoints based on the partially built scene representation, and therefore fail to effectively exploit the semantic and spatial information encoded within the graph during exploration. This paper presents a fully visual framework for the active, incremental construction of 3D scene graphs from RGB input only, addressing both limitations. The proposed approach unifies perception and planning around a shared structured representation that captures object semantics, 3D geometry, relational context, and information from multiple viewpoints. Because the framework is hardware-agnostic and relies only on RGB observations, it can incorporate inputs from both onboard robot cameras and fixed external cameras within the same representation. Experiments on the Replica dataset show that the RGB-only pipeline achieves F1-score parity with baselines using ground-truth depth. Active exploration experiments on ReplicaCAD further show that semantic-driven viewpoint selection detects more than twice as many objects as a geometric frontier-based baseline under the same exploration budget. Finally, the external-camera setting demonstrates that complementary RGB views can effectively bootstrap the scene graph and improve contextual understanding at no additional exploration cost. I Introduction Modern robotic systems operating in unstructured human environments—such as homes, hospitals, and offices—depend on rich contextual scene understanding to perceive, decide, and act autonomously [ 15 ] . Traditional perception pipelines either build low-level dense metric maps that capture free and occupied space but lack semantic context, or produce object-centric outputs such as bounding boxes and semantic segmentations, which isolate entities effectively yet fail to encode relational structure and distinguish semantically different but visually similar spatial configurations [ 27 ] . Reliable robot autonomy therefore requires structured scene representations that jointly capture which entities are present, where they are located in three-dimensional space, and how they relate to one another. To address these limitations, 3D scene graphs have emerged as a foundational architecture for environment perception and contextual reasoning [ 13 , 1 ] . A 3D scene graph models the environment as a graph whose nodes represent object instances with semantic labels and metric locations, while edges encode pairwise spatial and functional relationships. This compact, queryable representation bridges low-level visual perception and high-level decision-making, supporting language grounding and task planning [ 17 ] , navigation [ 25 ] , and manipulation in partially observed scenes [ 10 ] . Despite their theoretical utility, the deployment of 3D scene graphs is constrained by significant limitations. State-of-the-art frameworks typically require RGB-D cameras or LiDAR to acquire the dense depth required for metric 3D reconstruction [ 8 , 9 , 26 ] . This restricts deployment to specialized platforms, excluding scenarios where only commodity RGB cameras are available, such as surveillance infrastructure, low-cost robots, or handheld devices. Furthermore, existing 3D scene graph pipelines are predominantly passive, relying on pre-recorded observation trajectories rather than actively selecting informative viewpoints for efficient 3D scene graph construction, thus failing to fully exploit the rich information encoded within the graph. To address these limitations, we present a fully visual perception–action framework for active 3D scene graph construction from RGB images only, combining MapAnything-based feed-forward 3D reconstruction [ 11 ] , ConceptGraphs open-vocabulary semantic mapping [ 8 ] , and semantic exploration [ 23 ] within a single pipeline. Our contributions are: 1. an RGB-only pipeline for 3D scene graph generation; 2. the integration of this representation into an active semantic exploration loop for next-best-view selection; 3. an evaluation of fixed external RGB cameras as complementary observations for scene graph bootstrapping and improvement. II Related Work Fig. 1 : RGB-only active 3D scene graph generation loop. RGB images from the onboard camera (and, optionally, from fixed external cameras) feed the RGB-only pipeline, incrementally updating the point cloud and scene graph. The active module uses the current scene graph to select the NBV. The robot moves to the selected view and the loop repeats. 3D Scene Graphs (3DSGs). Scene graphs were introduced as structured representations coupling object semantics with relational context, addressing ambiguities in purely object-centric systems [ 27 ] . Extending this abstraction to 3D grounds entities and relationships in metric space, enabling navigation, manipulation, and language-guided planning [ 5 ] . Representative systems such as Hydra [ 9 ] and SceneGraphFusion [ 26 ] build hierarchical representations from SLAM outputs, relying on posed RGB-D data, but operate on passive trajectories and use closed-vocabulary semantics. Open-vocabulary methods, notably ConceptGraphs [ 8 ] , overcome vocabulary limits using foundation models for flexible language-grounded representations, yet still assume RGB-D inputs. Active scene graph construction remains underexplored: most approaches operate on fixed trajectories without influencing viewpoint selection [ 5 ] . Exceptions include the embodied framework of Li et al. [ 14 ] and Active Semantic Perception (ASP) [ 23 ] , which uses Large Language Model (LLM)-sampled scene graph completions to guide exploration, but still relies on depth observations and known camera poses. Active Perception. Active perception—controlling sensing to maximize information gain—has long been studied via Next-Best-View (NBV) planning [ 2 ] . Geometric frontier-based methods [ 28 ] and density-based approaches such as Surface Edge Explorer (SEE) [ 3 ] efficiently expand coverage but ignore semantics. Recent work incorporates semantic reasoning: Ding et al. [ 7 ] use semantic scene completion to guide exploration toward uncertain regions, while Chen et al. [ 6 ] combine semantic and geometric uncertainty via 3D Gaussian Splatting. ASP [ 23 ] further enables LLM-based reasoning over scene graphs, turning them into active exploration tools. However, prior work on ASP has not considered settings with external camera inputs, nor RGB-only perception. Multi-View Perception via External Cameras While robotic perception is typically egocentric, fixed external cameras provide complementary global context. Prior work shows their value for visual servoing [ 19 ] and VLM-guided navigation [ 4 ] , but their integration into 3DSGs remains unexplored. III Methodology III-A System Overview The system operates as an incremental perception–action loop (Fig. 1 ). At each step t , the robot captures an RGB image I_{t} from its current viewpoint v_{t}\in\mathcal{V} , where \mathcal{V} is the set of navigable viewpoints in the environment. The RGB-only 3DSG generation pipeline processes I_{t} to update the reconstructed point cloud \mathcal{P}_{t} and the scene graph \mathcal{G}_{t}=(\mathcal{O}_{t},\mathcal{E}_{t}) , where \mathcal{O}_{t} is the set of detected object nodes and \mathcal{E}_{t} the set of spatial relationship edges between them. The active exploration module then selects the Next-Best-View (NBV) v_{t+1} that maximizes expected information gain. The robot navigates to v_{t+1} and the loop repeats. Fixed external cameras inject additional RGB frames into the pipeline, contributing complementary viewpoints without robot motion. III-B RGB-only 3DSG generation Depths and Poses Estimation. The pipeline leverages MapAnything [

Read the complete text