COARSE: Collaborative Pseudo-Labeling with Coarse Real Labels for Off-Road Semantic Segmentation

Aurelio Noca, Xianmei Lei, Jonathan Becktor, Jeffrey Edlund, Anna Sabel, Patrick Spieler, Curtis Padgett, Alexandre Alahi, Deegan Atha · 2025

Read COARSE: Collaborative Pseudo-Labeling with Coarse Real Labels for Off-Road Semantic Segmentation by Aurelio Noca, Xianmei Lei, Jonathan Becktor, Jeffrey Edlund, Anna Sabel, Patrick Spieler, Curtis Padgett, Alexandre Alahi, Deegan Atha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03947.txt.

Introduction

Aurelio Noca and co-authors present COARSE, a way to teach off-road driving perception when detailed labels are rare. Self-driving vehicles in wild terrain must label mud, rock, brush, and track in every image, yet hand-labeling such scenes densely is scarce and costly. Simulation data help but look different from real trails.

Their method mixes sparse, rough real labels with dense labels from out-of-domain data. Two decoders, working at pixel and patch level, teach each other through shared pseudo-labels on unlabeled images. On two public off-road sets the scores jumped by roughly ten and eight points. Coarse human hints, used well, can substitute for expensive annotation campaigns.

Catalog record

CREATORAurelio Noca, Xianmei Lei, Jonathan Becktor, Jeffrey Edlund, Anna Sabel, Patrick Spieler, Curtis Padgett, Alexandre Alahi, Deegan Atha
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03947

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

COARSE: Collaborative Pseudo-Labeling with Coarse Real Labels for Off-Road Semantic Segmentation Aurelio Noca, Xianmei Lei, Jonathan Becktor, Jeffrey Edlund, Anna Sabel, Patrick Spieler, Curtis Padgett, Alexandre Alahi, Deegan Atha Abstract Autonomous off-road navigation faces challenges due to diverse, unstructured environments, requiring robust perception with both geometric and semantic understanding. However, scarce densely labeled semantic data limits generalization across domains. Simulated data helps, but introduces domain adaptation issues. We propose COARSE, a semi-supervised domain adaptation framework for off-road semantic segmentation, leveraging sparse, coarse in-domain labels and densely labeled out-of-domain data. Using pretrained vision transformers, we bridge domain gaps with complementary pixel-level and patch-level decoders, enhanced by a collaborative pseudo-labeling strategy on unlabeled data. Evaluations on RUGD and Rellis-3D datasets show significant improvements of 9.7% and 8.4% respectively, versus only using coarse data. Tests on real-world off-road vehicle data in a multi-biome setting further demonstrate COARSE’s applicability. I Introduction Off-road navigation presents itself as a challenge in many applications, including agricultural automation, disaster response and planetary exploration. Traversing unstructured and highly varied terrain – featuring unpredictable slopes, dense vegetation, and hidden obstacles – poses unique challenges that even skilled human drivers struggle to navigate reliably. Although standard approaches for mapping and geometric traversability analysis provide a framework for obstacle avoidance, they fail to distinguish between obstacles that are genuinely solid and those that appear solid but are actually traversable (e.g., tall grass, bushes, water puddles). Consequently, accurate semantic segmentation is a key requirement for safe and robust off-road autonomy. At the same time, the scarcity of densely labeled data in off-road domains limits the applicability of deep learning-based segmentation models. Producing large-scale pixel-wise annotations is laborious and expensive, especially when many environments exhibit ambiguous boundaries and overlapping vegetation classes. Recent work has thus moved toward semi-supervised learning (SSL) and domain adaptation techniques [ 1 , 2 ] , leveraging unlabeled data alongside sparse ground-truth labels. Moreover, recent vision foundation models [ 3 , 4 ] have demonstrated strong cross-domain generalization, mitigating some of the burden of in-domain labeling. In this paper, we introduce COARSE, a novel semi-supervised domain adaptation framework that fuses coarse in-domain labels with densely annotated out-of-domain data in a collaborative pseudo-labeling pipeline, featuring complementary pixel-level and patch-level decoders. We validate our approach on publicly available off-road datasets and demonstrate its effectiveness for real-world autonomous driving scenarios (see fig. 1 ). Fig. 1: Polaris RZR dune buggy, outfitted with an extensive sensor suite for autonomous off-road navigation, being driven in the San Diego Grasslands biome. Fig. 2: Our COARSE pseudo-labeling approach leverages two decoders – PatchDecoder and PixelDecoder – both utilizing the robust semantic features of the DINOv2 encoder. The PixelDecoder further integrates low-level geometric details from the input image. The PatchDecoder is trained on coarse ID data, while the PixelDecoder is trained on a combination of dense OOD and coarse ID data. Pseudo-labels are generated via disagreement of predicted semantic maps. II Related Work Early off-road segmentation methods often relied on handcrafted features such as color-space thresholding or texture filters to delineate vegetation from drivable surfaces [ 5 ] . Although these approaches worked well in controlled conditions, they generalized poorly to different terrains. With the success of convolutional neural networks (CNNs), fully convolutional architectures like FCN [ 6 ] and U-Net [ 7 ] paved the way for end-to-end semantic segmentation. More recent methods have incorporated attention-based models (e.g., Segmenter [ 8 ] , SegFormer [ 9 ] , DPT [ 10 ] , and Mask2Former [ 11 ] ), showing improved accuracy on dense labeling tasks. Additionally, investigations into ViTs models have demonstrated their resilience to distribution shifts and image perturbations like illuminations changes and noise [ 12 , 13 , 14 , 15 ] . However, obtaining extensive dense labels for off-road environments remains prohibitively expensive. Benchmarks such as RUGD [ 16 ] and Rellis-3D [ 17 ] illustrate large class imbalances and unclear boundaries, creating a strong need for data-efficient approaches. Semi-supervised learning (SSL) leverages large amounts of unlabeled data to compensate for limited annotations [ 1 , 2 , 18 ] , employing strategies like consistency regularization [ 19 ] or pseudo-labeling [ 20 ] . Meanwhile, domain adaptation methods tackle the shift between labeled source data (e.g., synthetic or out-of-distribution images) and unlabeled target data [ 21 , 22 ] . A key challenge in off-road segmentation is label scarcity and noise due to ambiguity, as many scenes can be coarsely or partially annotated. Recent work has shown that pretrained ViTs enable using coarse labels for few-shot semantic segmentation across multiple domains [ 23 ] . Alternatively, coarse labels can be refined via test-time augmentation [ 24 ] , or limited real data can be enhanced by synthetic data [ 25 , 26 , 27 , 28 , 29 ] . Further work has explored the use of a dual model architecture, a CNN and a Transformer, where each model’s output pseudo-labels are used to train the other [ 30 ] . Large pretrained vision encoders have emerged as a crucial building block in computer vision. Starting from earlier backbones like ResNet [ 31 ] or EfficientNet [ 32 ] , modern self-supervised models such as MAE [ 33 ] and DINO [ 3 ] demonstrate robustness and adaptability with minimal labeled data, culminating in DINOv2 [ 4 ] , which excels across a wide range of tasks. DINOv2 has further shown great promise in producing domain-independent features. By exploiting these pretrained features, an off-road segmentation pipeline can converge faster and achieve superior generalization. COARSE builds on these insights, using DINOv2 as a feature extractor followed by a complementary pair of decoders and large-scale unlabeled data to tackle off-road semantic segmentation with sparse real labels. III Method III-A Datasets Multi-Biome Real-World Dataset: For the challenge of off-road autonomous driving, we have collected data in a diverse set of locations, including the Mojave Desert, Paso Robles Grassland, San Diego Grassland, California Chapparal, and San Gabriel Canyon. The data collection strategy is the same as in [ 23 ] . This multi-biome dataset contains a large diversity of geological environments, as well as plants and water bodies. The data is collected in a ego-centric frame on a Polaris RZR vehicle. Then, 496 images are labeled with a density of less than 30%. Additionally, we have access to around 1000 unlabeled images from each location, totaling to 5000 unlabeled samples. In addition to the real images and labels, we also generated 1000 simulations samples using the tool from Duality AI [ 34 ] , simulating a sparse forest with tall grass, and obstacles like rocks and logs. The distribution of the simulation data differs from the real one in two aspects: the color distribution, and the class and instances distribution. Samples of the dataset are displayed in fig. 3 . Fig. 3: Samples (left) and labels (right) from our multi-biome dataset. Paso Robles Grassland (top), Mojave Desert (middle-top), San Gabriel Canyon (middle-bottom) and synthetic Forest-Sim (bottom). Open-Source Off-Road Datasets To quantify the effectiveness of our method, we consider two openly available off-road datasets: Rellis-3D [ 17 ] and RUGD [ 16 ] . The Rellis-3D data was collected at the Rellis Campus of Texas A&M University.

Read the complete text