ForestSim: A Synthetic Benchmark for Intelligent Vehicle Perception in Unstructured Forest Environments

Pragat Wagle, Zheng Chen, Lantao Liu · 2026

Read ForestSim: A Synthetic Benchmark for Intelligent Vehicle Perception in Unstructured Forest Environments by Pragat Wagle, Zheng Chen, Lantao Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27923.txt.

Introduction

Wagle and co-authors build ForestSim, a synthetic forest driving dataset. Urban datasets abound, but wild terrain data are scarce. Pixel-true labels in forests cost too much to hand-make. ForestSim holds 2,094 photorealistic images. They span twenty-five settings, seasons, and foliage levels. Unreal Engine plus AirSim generate the scenes. Twenty classes carry consistent labels. Top segmentation models benchmark well on the set.

Catalog record

CREATORPragat Wagle, Zheng Chen, Lantao Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26034
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27923

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ForestSim: A Synthetic Benchmark for Intelligent Vehicle Perception in Unstructured Forest Environments Pragat Wagle, Zheng Chen, Lantao Liu Abstract Robust scene understanding is essential for intelligent vehicles operating in natural, unstructured environments. While semantic segmentation datasets for structured urban driving are abundant, the datasets for extremely unstructured wild environments remain scarce due to the difficulty and cost of generating pixel-accurate annotations. These limitations hinder the development of perception systems needed for intelligent ground vehicles tasked with forestry automation, agricultural robotics, disaster response, and all-terrain mobility. To address this gap, we present ForestSim, a high-fidelity synthetic dataset designed for training and evaluating semantic segmentation models for intelligent vehicles in forested off-road and no-road environments. ForestSim contains 2094 photorealistic images across 25 diverse environments, covering multiple seasons, terrain types, and foliage densities. Using Unreal Engine environments integrated with Microsoft AirSim, we generate consistent, pixel-accurate labels across 20 classes relevant to autonomous navigation. We benchmark ForestSim using state-of-the-art architectures and report strong performance despite the inherent challenges of unstructured scenes. ForestSim provides a scalable and accessible foundation for perception research supporting the next generation of intelligent off-road vehicles. The dataset and code are publicly available: Dataset: https://vailforestsim.github.io Code: https://github.com/pragatwagle/ForestSim I Introduction Intelligent vehicles rely heavily on robust perception systems capable of interpreting diverse and dynamic environments. While significant progress has been made in urban autonomous driving supported by diverse, meticulously annotated datasets, achieving reliable autonomy beyond structured roads remains a major challenge. For example, many wild environments, including forests, agricultural fields, and natural terrain, present complexities such as irregular geometry, dense vegetation, indistinct object boundaries, and seasonal variations. These factors create substantial difficulty for image segmentation pipelines that intelligent vehicles depend on for environment understanding, obstacle avoidance, and traversability estimation. The advancement of computer vision systems is closely tied to the availability of large-scale image datasets. Such datasets have served as benchmarks and foundational resources for object detection, classification, and semantic segmentation, particularly when object classes are well represented in their environments [ 1 , 22 , 2 , 4 , 12 ] . The taxonomy of large datasets varies depending on their intrinsic characteristics. One popular taxonomy scheme categorizes image datasets into structured and unstructured datasets. Structured datasets, reminiscent of urban environments with clear boundaries, vehicular traffic, and regular-shaped buildings [ 23 , 8 , 11 , 38 , 27 ] , have been crucial in constructing high-performing semantic segmentation models. Furthermore, datasets are employed to augment existing models, enhancing the accuracy of semantic segmentation tasks. On the other hand, unstructured datasets exhibit significant variations in geometry, terrain, and appearance, presenting challenges such as navigational ambiguities in rough terrain and tall grass [ 20 , 35 , 26 , 3 , 18 ] . The creation of large pixelwise semantic labels poses substantial challenges, necessitating human intervention to ensure accuracy and quality. For instance, the CamVid dataset required 60 minutes per image for high-quality labeling [ 5 ] , while the Cityscapes dataset demanded 90 minutes per image [ 8 ] . Despite meticulous annotation efforts, generating pixel-accurate annotations often yields smaller datasets, particularly notable in high-quality semantic segmentation datasets. Beyond annotation challenges, navigating unstructured off-road environments presents additional hurdles, including resource scarcity and navigational complexities. This paper explores the utilization of commercial environments built using Unreal Engine to generate pixel-accurate ground truth data for training semantic segmentation models. Leveraging the abundance of environments available within Unreal Engine, ranging in scale, coupled with the Microsoft-owned open-source tool AirSim, allows for the collection of unprocessed semantic segmentation images with persistent random labels across different instances. The ForestSim dataset, introduced herein, aims to enhance the accuracy of semantic segmentation models across various terrain types, leveraging diverse environments representative of different seasons. The dataset comprises 2094 RGB images with corresponding pixel-wise ground truth annotations, extracted from 25 different high-quality, realistic environments. Although ForestSim is smaller in scale its focus on unstructured forested seasonal environments with pixel accurate annotation provides a complimentary resource for studying off-road perception challenges. An example RGB image and ground truth pixel-level semantic segmentation image produced from the data collected from Unreal Engine are illustrated in Fig. . Benchmarks, employing up-to-date methodologies such as Mean Intersection-Over-Union and Pixel Accuracy metrics [ 35 , 18 ] , validate the efficacy of the proposed dataset. We introduce this dataset providing pixel-accurate semantic labels focusing exclusively on densely vegetated forested, seasonal, and off-road scenes and establish baseline performance with the goal of enhancing the capabilities of existing machine learning models. By leveraging this dataset, we see potential to support autonomous field robots in a myriad of tasks, including timber sorting, harvesting operations, agricultural field tasks, and surveillance in challenging, unstructured environments. We leave environment-disjoint and cross-environment evaluation as future work. II Segmentation Datasets Semantic segmentation datasets serve as foundational resources for partitioning images into meaningful parts through pixel-wise annotation. This segmentation task is fundamental for various applications in computer vision, facilitating tasks such as object detection, scene understanding, and autonomous navigation [ 13 ] . These pipelines include utilizing an encoder to create hierarchical representations of an image using a backbone such as ResNet with a decoder that upsamples samples through convolutional layers, converting low dimensional features to original resolutions, which are the feature maps that are eventually used for pixel-wise prediction. II-A Structured Datasets Structured semantic segmentation datasets represent environments with well-defined boundaries and organized elements. This category encompasses a plethora of datasets, including COCO-Stuff [ 23 ] , Pascal VOC [ 11 ] , ADE20K [ 38 ] , Pascal Context [ 27 ] , Audi Structured [ 15 ] , Cityscapes [ 8 ] , KITTI [ 14 ] , Mapillary [ 10 ] , and ApolloScape [ 17 ] . Among numerous selections, for example, Mapillary provides a benchmark dataset specifically tailored for traffic sign classification [ 10 ] , while KITTI focuses on common urban objects like buildings, trees, cars, and roads [ 14 ] . ApolloScape offers a diverse range of data captured from various cities and different times of the day, integrating camera videos, consumer-grade motion sensors, and 3D semantic maps [ 17 ] . These datasets are often collected by involving ground vehicles equipped with multiple sensors, capturing rich data from urban environments [ 8 , 17 , 14 ] . II-B Unstructured Datasets In contrast, unstructured semantic segmentation datasets capture environments with complex and varied characteristics, including rugged terrain, dense vegetation, and irregular structures. The RUGD dataset [ 35 ] serves as a benchmark for unstructured environments near creeks, vegetation, water bodies, trails, and villages. TAS500 [ 26 ] focuses on discerning traversable regions from non-traversable ones, categorizing 44 different objects into nine groups. The Rellis dataset [ 18 ] comprises synchronized sensor data collected using a mobile robotic platform, featuring diverse terrains like runways, aprons, and lakes. These datasets play a crucial role in advancing the robustness and adaptability of semantic segmentation models in challenging real-world scenarios. However, compared to the structured datasets, the number of unstructured datasets is significantly lower. This scarcity stresses the importance of creating more datasets in this category to provide a more comprehensive representation of diverse and challenging environments. In our work, we provide a very accessible and reusable process

Read the complete text