Uncertainty evaluation of segmentation models for Earth observation

Mélanie Rey, Andriy Mnih, Maxim Neumann, Matt Overlan, Drew Purves · 2025

Read Uncertainty evaluation of segmentation models for Earth observation by Mélanie Rey, Andriy Mnih, Maxim Neumann, Matt Overlan, Drew Purves (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19586.txt.

Introduction

Melanie Rey and four co-authors benchmark ways to say how unsure an image segmentation model is, for models that label satellite photos pixel by pixel. Pixel-level doubt scores must scale to whole scenes, which makes the task harder than for simple image sorting. Most past work on model doubt targeted street scenes or medical scans, not Earth data.

The tests check how well the doubt scores flag real errors and corrupted parts of an input image. Two datasets are used, PASTIS and ForTy, picked for their unlike scales, regions, and label quality. Several model types, among them randomized segmentation networks and ensembles, are paired with varied network designs and doubt metrics, and the paper ends with advice for users.

Catalog record

CREATORMélanie Rey, Andriy Mnih, Maxim Neumann, Matt Overlan, Drew Purves
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uncertainty evaluation of segmentation models for Earth observation Mélanie Rey, Andriy Mnih, Maxim Neumann, Matt Overlan, Drew Purves Abstract This paper investigates methods for estimating uncertainty in semantic segmentation predictions derived from satellite imagery. Estimating uncertainty for segmentation presents unique challenges compared to standard image classification, requiring scalable methods producing per-pixel estimates. While most research on this topic has focused on scene understanding or medical imaging, this work benchmarks existing methods specifically for remote sensing and Earth observation applications. Our evaluation focuses on the practical utility of uncertainty measures, testing their ability to identify prediction errors and noise-corrupted input image regions. Experiments are conducted on two remote sensing datasets, PASTIS and ForTy, selected for their differences in scale, geographic coverage, and label confidence. We perform an extensive evaluation featuring several models, such as Stochastic Segmentation Networks and ensembles, in combination with a number of neural architectures and uncertainty metrics. We make a number of practical recommendations based on our findings. 1 Introduction Estimating prediction uncertainty for deep learning models is a complex yet crucial task for their application in complex, real-world domains. Uncertainty quantification is important for two primary reasons: first, some fields, such as medical science, require high-confidence predictions for practical use, and second, quantifying uncertainty offers a powerful method for improving models by highlighting their weaknesses and potential data issues. Uncertainty estimation for remote sensing and Earth observation has gained momentum recently ( Singh et al. 2024 ; Lang et al. 2022 ) ; however, a systematic evaluation and comparison across different architectures and datasets is still lacking. There are two main types of uncertainty to consider when making predictions using a model. The aleatoric uncertainty corresponds to the noise inherent to the data-generating process when the output is not a deterministic function of the input; this is orthogonal to any modeling considerations. The epistemic uncertainty is the uncertainty about the model class and its parameter setting used to produce the outputs from the inputs; it is sometimes referred to as the model uncertainty. In this work, we do not aim to quantify these uncertainty types separately, as doing so has been shown to be difficult in practice ( Kahl et al. 2024 ) . Instead, we focus on metrics that evaluate the overall uncertainty and answer pragmatic questions for practitioners: (1) Can we detect when the model will make inaccurate predictions? (2) Can we identify corrupted data? (3) Can the model’s performance be improved through (post-hoc) uncertainty estimation? While these questions do not encompass the full scope of what uncertainty should ideally capture (e.g., ambiguous examples or input distribution shifts), they provide a solid starting point for exploring model uncertainty in practical scenarios. Our analysis leads to the following main findings: • Common uncertainty estimation methods demonstrate limited performance in the task of identifying misclassified pixels at test time. However, these methods are substantially more effective at identifying poorly segmented images when evaluated at the image level. • The choice of model architecture significantly impacts uncertainty estimation; we find that Vision Transformer-based models substantially outperform convolutional models at identifying segmentation errors. • Since performance varies across architectures and datasets, we recommend the practice of setting aside a dedicated “uncertainty test set” for robust evaluation. • We show that Stochastic Segmentation Networks (SSN), which use a Gaussian latent layer to model correlation between labels at different locations, work well with the Transformer architecture. They deliver improved segmentation performance in both the noise-free setting and when training with noisy inputs, compared to the standard Transformer-based models without latent variables. 2 Related work Estimating the uncertainty of deep learning model predictions is an active area of research, particularly in probability calibration and uncertainty modeling. The first research stream investigates whether the confidence scores produced by a model, typically the output probabilities from a softmax layer, are calibrated, meaning they accurately reflect the true likelihood of correctness. Early work focused on developing reliable metrics for measuring calibration and exploring the impact of model architecture and capacity on miscalibration ( Guo et al. 2017 ; Nixon et al. 2020 ; Minderer et al. 2021 ) . Subsequent studies have also examined the challenge of maintaining calibration under dataset distribution shifts ( Ovadia et al. 2019 ; Cygert et al. 2021 ) . A variety of post-hoc and in-training methods have been proposed to improve model calibration ( Kumar et al. 2019 ; Mukhoti et al. 2020 ; Kull et al. 2019 ) . The second stream, often inspired by Bayesian principles, focuses on directly modeling a model’s uncertainty. Seminal methods in this category include Monte Carlo (MC) Dropout ( Gal and Ghahramani 2016 ) , which casts Dropout training as approximates Bayesian inference, and Deep Ensembles ( Lakshminarayanan et al. 2017 ) , which leverage the diversity in ensemble member predictions to estimate uncertainty. Extending these concepts to semantic segmentation introduces additional complexities, as uncertainty is no longer a single value for an image but a dense, pixel-wise map where spatial correlations are important. Kendall and Gal 2017 were among the first to systematically tackle this, adapting uncertainty techniques for dense prediction tasks. Much of the foundational work in this area has been conducted in the domain of medical imaging, where quantifying uncertainty is critical for clinical decision-making. Researchers in this field have successfully applied a range of techniques, including Bayesian neural networks, ensembles, and MC Dropout, to tasks like tumor and organ segmentation ( Ng et al. 2023a ; Hoebel et al. 2020 ; Kwon et al. 2020 ) . The specific challenge of calibrating segmentation models has also been addressed, with studies investigating the impact of training procedures like the Dice loss ( Mehrtash et al. 2019 ) and introducing new methods like selective scaling tailored for segmentation outputs ( Wang et al. 2023 ) . Despite active research on uncertainty estimation methods ( Holder et al. 2021 ; Obster et al. 2024 ; Maag and Riedlinger 2024 ) , comprehensive benchmarks evaluating uncertainty estimation for segmentation are relatively rare. Notable exceptions include Ng et al. 2023b , who evaluated MC Dropout, Deep Ensembles, and other methods on cardiac MRI datasets, and Aerts et al. 2020 , who explored different uncertainty quantification methods at both the pixel and image level for lung nodule segmentation. More recently, Buchanan et al. 2022 provided an extensive benchmark across a variety of datasets, and Kahl et al. 2024 proposed a structured framework for evaluating the practical value of uncertainty in segmentation based on predominant uncertainty applications such as failure detection and out-of-distribution detection. Whereas Kahl et al. 2024 focuses on image-level uncertainty evaluation, we consider pixel-wise metrics as a local spatial analysis is desirable in many remote sensing applications. In the domain of remote sensing, uncertainty estimation is an emerging but not yet standard practice. While some recent work has focused on related problems like out-of-distribution detection ( Ekim et al. 2025 ) , bias in map-derived regression coefficients ( Lu et al. 2025 ) or ensemble learning for regression tasks ( Lang et al. 2022 ; Lang et al. 2023 ) , research on segmentation-specific uncertainty remains less common. Existing studies have explored Bayesian methods ( Haas and Rabus 2021 ; Dechesne et

Read the complete text