Modernising Reinforcement Learning–Based Navigation for Embodied Semantic Scene Graph Generation
Roman Küble, Marco Hüller, Mrunmai Phatak, Rainer Lienhart, Jörg Hähner · 2026
Read Modernising Reinforcement Learning–Based Navigation for Embodied Semantic Scene Graph Generation by Roman Küble, Marco Hüller, Mrunmai Phatak, Rainer Lienhart, Jörg Hähner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25415.txt.
Introduction
Küble and co-authors refresh how a robot explores to build a semantic map of a room. The map stores objects, their links, and where they sit. With few moves to spare, the robot must trade new facts against travel cost. Their module swaps in a modern policy optimizer and rethinks the move set. Fine, split-head moves beat one coarse choice. Just the optimizer swap lifts map completeness by about a fifth. Depth sensing mainly guards against crashes. The best mix pairs both changes.
Catalog record
| CREATOR | Roman Küble, Marco Hüller, Mrunmai Phatak, Rainer Lienhart, Jörg Hähner |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68573 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25415
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modernising Reinforcement Learning–Based Navigation for Embodied Semantic Scene Graph Generation Roman Küble, Marco Hüller, Mrunmai Phatak, Rainer Lienhart, Jörg Hähner Abstract Semantic world models enable embodied agents to reason about objects, relations, and spatial context beyond purely geometric representations. In Organic Computing, such models are a key enabler for objective-driven self-adaptation under uncertainty and resource constraints. The core challenge is to acquire observations maximising model quality and downstream usefulness within a limited action budget. Semantic scene graphs (SSGs) provide a structured and compact representation for this purpose. However, constructing them within a finite action horizon requires exploration strategies that trade off information gain against navigation cost and decide when additional actions yield diminishing returns. This work presents a modular navigation component for Embodied Semantic Scene Graph Generation and modernises its decision-making by replacing the policy-optimisation method and revisiting the discrete action formulation. We study compact and finer-grained, larger discrete motion sets and compare a single-head policy over atomic actions with a factorised multi-head policy over action components. We evaluate curriculum learning and optional depth-based collision supervision, and assess SSG completeness, execution safety, and navigation behaviour. Results show that replacing the optimisation algorithm alone improves SSG completeness by 21% relative to the baseline under identical reward shaping. Depth mainly affects execution safety (collision-free motion), while completeness remains largely unchanged. Combining modern optimisation with a finer-grained, factorised action representation yields the strongest overall completeness–efficiency trade-off. 1 Introduction The ability of autonomous, embodied systems [ 6 ] to create semantically rich world models in unknown, dynamic environments is a core requirement for intelligent behaviour. From the perspective of Organic Computing (OC) [ 22 , 16 , 15 ] , this represents a fundamental challenge of self-optimisation under uncertainty: An agent must decide how to use its limited action budget to learn a policy beneficial for downstream tasks. The efficiency of this process is largely determined by the system’s ability to balance exploration and knowledge gain and to autonomously adapt its own navigation behaviour to the conditions of the environment. The construction of such models serves as the necessary knowledge base for future OC-style self-X properties such as self-configuration or self-healing . A reliable representation of the environment is a prerequisite for the system to independently detect deviations and initiate appropriate corrective measures. Semantic Scene Graphs (SSGs) [ 4 ] have established themselves as a particularly suitable representation for this purpose. In contrast to purely geometric maps, SSGs organise objects, their attributes, and their semantic relations in a compact, graph-based structure [ 8 ] . This enables reasoning about object identity and relations in addition to location. However, constructing such graphs in real time requires targeted exploration and viewpoint control, a problem formalised by Li et al. [ 13 ] as Embodied Semantic Scene Graph Generation (ESSG). In ESSG, an embodied agent explores an unknown scene from egocentric viewpoints and incrementally builds a semantic scene graph under a fixed action budget. The pioneering approach for ESSG by Li et al. [ 13 ] formulates graph construction as a Reinforcement Learning (RL) problem, in which the agent is rewarded for discovering new objects and relations. However, the approach has major scalability and stability limitations: The use of the REINFORCE algorithm [ 24 ] often leads to unstable learning behaviour, while the severely restricted action discretisation limits the efficiency of exploration in complex environments. Moreover, navigation is not treated as an explicitly optimisable component that can adapt to different utility signals. In this paper, we address these limitations by presenting a modernised navigation component for ESSG, where navigation is understood as viewpoint selection for maximising scene-graph completeness. Rather than proposing a new end-to-end pipeline, we isolate navigation as a self-optimising component under a fixed action budget, motivated by OC principles. This abstraction lets us study how navigation choices affect scene-graph completeness and provides groundwork for future OC-style self-X mechanisms built on reliable semantic world models. Concretely, we replace REINFORCE with Proximal Policy Optimization (PPO) [ 19 ] under identical reward shaping. We then compare compact and larger, finer-grained discrete motion sets as well as a single-head policy over atomic actions (SH) and a multi-head factorised policy over action components (MH). Factorised means the policy chooses rotation, step length, and stop decision separately rather than selecting one monolithic action ID. In addition, we evaluate curriculum learning (CL) [ 17 ] , i.e. a staged progression from simpler to more difficult control settings, and optional depth-based collision supervision. By using an explicit Stop action, we enable autonomous termination that trades off information gain against budget usage. We make the following contributions: • We show that modern policy optimisation improves SSG completeness under identical reward shaping. • We characterise how discrete motion expressiveness affects completeness and behaviour under a fixed action horizon. • We demonstrate that factorised high-resolution control improves optimisation behaviour and safety diagnostics over an equivalent atomic action set while maintaining comparable or higher completeness. • We analyse how CL and depth-based supervision affect learning and execution across action regimes. The remainder of this paper is structured as follows: Section 2 reviews relevant work on semantic exploration and SSG-based world models. Section 3 defines the state representation, action-space design, reward shaping, and policy architecture. We then present the experimental design in Section 4 and the results in Section 5 , discussing their implications for learning-based semantic exploration. The paper concludes with a summary and an outlook on future research directions in Section 6 . All components of the experimental setup, the complete training pipeline, and the paper’s appendix are publicly available in an accompanying GitHub repository 1 1 1 https://github.com/kueblero/Modernising-RL-Based-Navigation-for-ESSG , enabling reproducibility and supporting future research efforts. 2 Related Work The construction of semantic models poses a central challenge for autonomous exploration, with SSGs proving particularly suitable for capturing objects, their attributes, and semantic relations in a unified structure [ 8 ] . In the existing literature, these representations are often constructed incrementally during the execution of specific tasks. For example, Tan et al. [ 20 ] and Loo et al. [ 14 ] present approaches in which SSGs are constructed “on the fly” to solve complex goals such as searching for objects or answering visual questions in unknown environments. In these works, however, exploration primarily serves as a secondary objective, while the optimisation of the navigation policy is focused on the respective downstream task. In contrast, strategies for explicit semantic exploration emphasise model coverage and completeness. Classically, heuristic or geometrically motivated methods such as frontier exploration [ 25 ] or Next-Best-View methods (NBV) [ 2 ] are used for this purpose. Lang et al. [ 12 ] present a current hybrid approach with SELM ( Semantic Exploration and Long-term Monitoring ). They establish the completeness of the SSG ( 3-DSG completeness ) as an explicit goal and use a combination of NBV planning and semantic heuristics to achieve efficient space coverage. While SELM uses RL only for downstream monitoring tasks, the exploration itself is heuristically controlled and thus relies on predefined geometric rules. Beyond purely geometric approaches, work in the field of Active Neural SLAM [ 5 ] shows that learning-based exploration strategies are capable of learning structural regularities of buildings and thus generating more