Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data
Marcus G Müller, Wout Boerdijk, Maximilian Durner, Riccardo Giubilato, Abel Gawel, Wolfgang Stürzl, Roland Siegwart, Rudolph Triebel · 2026
Read Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data by Marcus G Müller, Wout Boerdijk, Maximilian Durner, Riccardo Giubilato, Abel Gawel, Wolfgang Stürzl, Roland Siegwart, Rudolph Triebel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27644.txt.
Introduction
Marcus Muller and colleagues unify class-specific semantic segmentation with class-agnostic terrain segmentation for outdoor robots. Their Trinity network segments terrain from visual appearance alone, without preset labels or robot-dependent movement scores, learning shared visual terrain priors. Training extends the OAISYS simulator with a synthetic RUGDSynth dataset, plus a real-world EXTerra dataset carrying both label types.
Catalog record
| CREATOR | Marcus G Müller, Wout Boerdijk, Maximilian Durner, Riccardo Giubilato, Abel Gawel, Wolfgang Stürzl, Roland Siegwart, Rudolph Triebel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30567 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27644
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data Marcus G Müller, Wout Boerdijk, Maximilian Durner, Riccardo Giubilato, Abel Gawel, Wolfgang Stürzl, Roland Siegwart, Rudolph Triebel Abstract Terrain understanding is fundamental for mobile robots operating in unstructured outdoor environments. Existing vision-based traversability estimation methods rely on robot-specific annotations or semantic class mappings, limiting transferability across platforms and requiring costly re-annotation when robot capabilities change, while standard semantic segmentation methods only focus on specific predefined classes, which do not capture the variety of terrains. In this work, we propose a transformer-based architecture that jointly performs class-specific semantic segmentation and class-agnostic terrain segmentation within a unified network, called Trinity. Terrain regions are segmented based solely on visual appearance, without predefined semantic labels or robot-dependent traversability scores. This formulation enables the learning of robot-agnostic visual terrain priors that can be combined with robot-specific experience for downstream tasks such as traversability estimation, visual odometry, and mission planning. To enable large-scale training with diverse terrain appearances, we extend the OAISYS simulator and introduce RUGDSynth, a synthetic dataset inspired by RUGD with class-agnostic terrain samples. Furthermore, we present the EXTerra Dataset, providing real-world images annotated with both class-specific and class-agnostic terrain labels. Experiments demonstrate the feasibility of the proposed task and the effectiveness of our joint segmentation approach in complex outdoor environments. Code and datasets will be released with this publication (after review). I Introduction Semantic scene understanding is a fundamental capability for mobile robots operating in complex outdoor environments. In such settings, reliable perception supports navigation, state estimation, path planning, and manipulation. It can further enable higher-level objectives such as identifying scientifically relevant areas or suitable terrain for infrastructure deployment in planetary exploration. A central component of outdoor scene understanding is terrain segmentation, with traversability estimation as a closely related task. While supervised semantic segmentation has achieved remarkable success in structured urban environments [ 35 ] , transferring these approaches to natural terrain remains challenging. In contrast to roads or buildings with well-defined geometry, outdoor environments exhibit irregular regions, gradual transitions, and visually similar surface types. Boundaries are often ambiguous, and semantic categories are less clearly separable, making also the annotation process difficult [ 32 ] . Fig. 1: A rover and a drone observe a planetary exploration outdoor laboratory. The overlapping annotations shown on the color image illustrate the ideal segmentation results produced by the proposed method. The task is to segment different terrain types (shown in shades of cyan) in a class-agnostic manner, enabling their use across a wide range of robotic applications and platforms, while simultaneously identifying class-specific regions that are important for a mission (here: rock, fence, and tree). Beyond these perceptual challenges, terrain semantics are inherently context-dependent. The appearance and meaning of concepts such as soil or grass vary with environmental conditions (e.g., weather, season, illumination) and with the operational context of the robot. A compact dirt path may be easily traversable for a wheeled platform in dry conditions, yet become impassable mud after rainfall; dense grass may pose no difficulty for a tracked vehicle but obstruct a small wheeled robot. Consequently, semantic labels and especially traversability assessments are not absolute properties, but depend on both environmental and robot-specific factors. This variability makes it difficult to define a universally valid, fixed taxonomy of terrain classes. Most existing methods, however, assume precisely such predefined class sets and operate under a closed-world assumption in which all relevant categories are known during training. In traversability estimation, supervision is further tied to a specific robot platform, since annotations implicitly encode its mobility capabilities. As a result, deploying models in new environments or on different robots typically requires new data collection and retraining, limiting scalability and robustness. To address these challenges, we introduce Trinity-Net, a unified transformer-based architecture for flexible scene understanding that jointly performs class-agnostic and class-specific pixel-wise segmentation without relying on fixed terrain taxonomy. The class-agnostic part separates scenes into visually coherent terrain regions without predefined semantic labels. By focusing purely on appearance, it provides a robot-agnostic terrain representation that can serve as a visual prior for (self-supervised) downstream tasks [ 11 , 17 ] . Unlike local region-proposal methods [ 19 ] , Trinity-Net is able to enforce global visual consistency across spatially disjoint terrain regions. The class-specific branch predicts platform- and context-independent or mission critical semantic categories (e.g., trees, rocks, buildings). This separation allows the model to retain stable semantic anchors while avoiding an overconstrained terrain taxonomy. Fig. 1 gives an illustration of the overall task. Training such a system requires substantial terrain diversity to ensure robust generalization and genuinely class-agnostic behavior. However, annotating large-scale datasets with terrain labels is costly and time-consuming. To address this limitation, we additionally leverage synthetic data. Unfortunately, many existing outdoor datasets do not provide terrain-specific annotations. Moreover, most simulation environments are unable to output such information, as their semantic labels are typically object-based rather than texture- or material-based. In contrast, the OAISYS simulator [ 24 ] provides semantic annotations using a material-based labeling approach. This enables the extraction of terrain annotations corresponding to different surface textures. To this end, we leverage and extend the OAISYS simulator to generate a large-scale synthetic dataset for joint training. Our approach achieves strong performance on the RUGD benchmark [ 37 ] across both segmentation tasks. Furthermore, we demonstrate transferability by evaluating the trained model—developed in a field robotics context—on a real-world dataset recorded for planetary exploration scenarios. To summarize, our contributions are: (i) Trinity-Net, a unified transformer architecture for joint class-specific semantic and class-agnostic segmentation, trained on synthetic data to learn transferable appearance-based terrain representations; (ii) an extension of the OAISYS simulator enabling large-scale synthetic data generation for this task, resulting in RUGDSynth , a synthetic counterpart to the field-robotics RUGD benchmark, and (iii) the EXTerra Dataset , a real-world planetary exploration dataset from an analog mission site. II Related Work Our work builds on semantic segmentation [ 7 , 33 ] , with emphasis on terrain segmentation, traversability estimation, and open-world segmentation. Terrain Segmentation is commonly formulated as supervised semantic segmentation with predefined terrain taxonomies, requiring extensive manual annotation and fixed class definitions. These limitations become particularly evident in planetary exploration, where annotated data is scarce and highly variable terrain appearance challenges rigid semantic categories. Existing approaches range from rock-focused obstacle classification [ 8 , 23 ] to broader supervised semantic labeling efforts [ 32 , 14 ] . To improve robustness multimodal perception is additionally explored, by fusing visual and thermal imagery [ 4 ] . Several works also investigate reduced-supervision, including sparse labeling [ 13 ] , semi-supervised learning [ 39 ] , and text-guided foundation-model-based segmentation [ 10 ] . Despite mitigating annotation scarcity, these methods still operate within predefined terrain categories. Similar reliance on supervision and fixed semantic taxonomies appears in terrestrial unstructured outdoor environments. For instance, CRLNet [ 21 ] combines convolutional and attention mechanisms in a supervised setting, reporting strong results on RUGD [ 37 ] and RELLIS-3D [ 16 ] . Other approaches reduce semantic labels to navigability-oriented categories. In [ 15 ] a multi-scale transformer to aggregate contextual features and cluster terrains into roughness-based groups for navigation is employed. Similarly, Li et al. [