Ensemble-Based Dirichlet Modeling for Predictive Uncertainty and Selective Classification

Courtney Franzen, Farhad Pourkamali-Anaraki · 2026

Read Ensemble-Based Dirichlet Modeling for Predictive Uncertainty and Selective Classification by Courtney Franzen, Farhad Pourkamali-Anaraki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06032.txt.

Introduction

Franzen and Pourkamali-Anaraki's 2026 paper tackles uncertainty in neural network classifiers. Their softmax scores give no built-in trust measure and swing across training runs. Instead of fragile evidential training, they estimate Dirichlet parameters from ensembles of softmax outputs. They use a method-of-moments estimator, with an optional maximum-likelihood refinement. The build separates uncertainty work from touchy loss design and steadies single-run swings. Across several datasets, the ensemble-derived estimates improve confidence scores and selective classification, where a model holds back on shaky calls.

Catalog record

CREATORCourtney Franzen, Farhad Pourkamali-Anaraki
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89257
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06032

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ensemble-Based Dirichlet Modeling for Predictive Uncertainty and Selective Classification Courtney Franzen, Farhad Pourkamali-Anaraki Abstract Neural network classifiers trained with cross-entropy loss achieve strong predictive accuracy but lack the capability to provide inherent predictive uncertainty estimates, thus requiring external techniques to obtain these estimates. In addition, softmax scores for the true class can vary substantially across independent training runs, which limits the reliability of uncertainty-based decisions in downstream tasks. Evidential Deep Learning aims to address these limitations by producing uncertainty estimates in a single pass, but evidential training is highly sensitive to design choices including loss formulation, prior regularization, and activation functions. Therefore, this work introduces an alternative Dirichlet parameter estimation strategy by applying a method of moments estimator to ensembles of softmax outputs, with an optional maximum-likelihood refinement step. This ensemble-based construction decouples uncertainty estimation from the fragile evidential loss design while also mitigating the variability of single-run cross-entropy training, producing explicit Dirichlet predictive distributions. Across multiple datasets, we show that the improved stability and predictive uncertainty behavior of these ensemble-derived Dirichlet estimates translate into stronger performance in downstream uncertainty-guided applications such as prediction confidence scoring and selective classification. keywords Dirichlet Modeling, Method of Moments, Selective Classification, Predictive Uncertainty, Cross-Entropy Variability 1 Introduction Deploying classifiers in high-stakes domains requires principled estimates of predictive uncertainty to support reliable decision making. In standard classification systems trained with cross-entropy (CE) loss, the output consists of softmax scores rather than explicit uncertainty estimates. Although these scores are often interpreted as measures of predictive confidence, they are not guaranteed to reflect calibrated correctness and do not provide explicit measures of uncertainty Guo et al. 2017 ; Tomani and Buettner 2021 ; Molchanov et al. 2017 ; Kingma et al. 2015 ; Gal and Ghahramani 2016 . Moreover, the magnitude of softmax scores can vary substantially across independent training runs, even when predicted class labels remain unchanged Lakshminarayanan et al. 2017 ; Ashukha et al. 2021 ; Monarch 2021 . Fig. 1 illustrates this phenomenon on two representative samples from a 10-class image dataset, where identical inputs yield noticeably different probability estimates under different random initializations. Taken together, these limitations highlight the need for predictive representations that encode uncertainty more directly and consistently than single-run softmax outputs. Evidential Deep Learning (EDL) aims to address this need by modeling class probabilities as a Dirichlet-distributed random variable ( Sensoy et al. 2018 ; Ulmer et al. 2023 ) . Grounded in Subjective Logic ( Jøsang and service) 2016 ) , the approach interprets the Dirichlet distribution as representing uncertainty over categorical probabilities relative to a fixed reference distribution (the base rate ), which encodes the prior probability of each class in the absence of evidence. Rather than introducing parameter stochasticity to capture predictive uncertainty, an EDL network deterministically outputs nonnegative evidence values that define Dirichlet concentration parameters for each input. These concentration parameters specify a predictive distribution over categorical likelihoods, yielding both a predictive mean and an associated variance in a single pass. In this formulation, uncertainty is encoded directly in the predictive distribution itself. Despite its theoretical appeal, EDL suffers from practical challenges. Its performance is sensitive to design choices such as loss function, prior regularization, and evidence activation Sensoy et al. 2021 ; Jürgens et al. 2024 ; Davies et al. 2023 ; Shen et al. 2024 ; Malinin and Gales 2018 . These choices can dramatically affect the Dirichlet concentration parameters and, consequently, the resulting predictive variance. Some configurations lead to unstable training, collapsed uncertainty, or unreliable outputs, particularly in fine-grained or high-class-count tasks. These issues limit the robustness and generalizability of EDL across datasets Pandey et al. 2025 , motivating the need for fallback or complementary methods when evidential training fails. Figure 1 : Frequency plots of softmax scores for the true class for two representative samples from a 10-class image classification task across 100 randomly initialized classifiers trained with a cross-entropy loss function. The left plot (automobile) shows a difficult sample with highly variable scores, while the right plot (deer) shows an easier sample whose scores remain concentrated but still exhibit nontrivial variability. To address this, we propose a simple and effective alternative for generating Dirichlet-based uncertainty estimates: a method of moments (MoM) construction with optional maximum likelihood refinement that fits a Dirichlet distribution to the empirical distribution of softmax outputs from an ensemble of CE-trained models. Fig. 2 shows how repeated softmax vectors for a fixed input can be treated as samples on the probability simplex. By matching class-wise empirical means and variances to a Dirichlet distribution, we obtain per-input concentration parameters that encode input-dependent uncertainty without requiring evidential training. This MoM-derived Dirichlet construction produces predictive distributions that match EDL in interpretability and application, enabling direct comparison in downstream tasks such as calibration, confidence ranking, and selective classification. In particular, we show that total Dirichlet variance can serve as an informative ordering statistic for abstention, and that MoM-derived models can achieve superior performance on variance-based selective classification metrics compared to EDL configurations. In total, this work makes the following contributions: 1. We introduce a method of moments estimator for constructing Dirichlet distributions from ensembles of CE–trained models, yielding input-dependent concentration parameters without specialized evidential loss design or unstable training dynamics. 2. We investigate an optional maximum likelihood refinement via fixed-point iteration, evaluating whether likelihood-based optimization meaningfully improves moment-based Dirichlet estimates in calibration and selective classification settings. 3. We provide a systematic empirical analysis of Dirichlet-based classifiers for uncertainty-aware decision making, showing that evidential design choices induce substantial variation in confidence behavior, calibration diagnostics, and selective performance across otherwise similar models. 4. We empirically demonstrate that ensemble-derived Dirichlet variance serves as an effective ordering statistic for selective classification, achieving competitive or superior retained accuracy, F1 score, and negative log-likelihood under matched risk constraints Figure 2 : Overview of the method of moments Dirichlet construction. An ensemble of CE-trained models produces multiple softmax probability vectors for a fixed input. These are treated as empirical samples on the probability simplex, from which class-wise means and variances are computed and matched to a Dirichlet distribution, yielding per-input concentration parameters \hat{\boldsymbol{\alpha}} . The remainder of the paper is organized as follows. Section 2 reviews Dirichlet-based uncertainty modeling and confidence calibration analysis. Section 3 presents the ensemble-based Dirichlet construction and its optional maximum likelihood refinement, followed by the variance-based selective classification framework. Section 4 reports three experimental studies: calibration analysis across evidential formulations, comparison of moment-based and likelihood-refined Dirichlet estimators against evidential baselines, and a variance-based selective classification study evaluating risk–coverage performance under fixed risk levels. Finally, Section 5 discusses implications for uncertainty modeling and outlines directions for future work. 2 Background 2.1 Evidential Deep Learning In classification tasks, neural networks typically produce classification scores using a final linear layer defined as \mathbf{z}:=\mathbf{W}\mathbf{h}+\mathbf{b}\in\mathbb{R}^{K}, (1) where \mathbf{h}\in\mathbb{R}^{d} is the final hidden layer feature vector, \mathbf{W}\in\mathbb{R}^{K\times d} is the output weight matrix, and \mathbf{b}\in\mathbb{R}^{K} is the bias vector. These logits \mathbf{z} are then transformed (via a model-specific mapping) into a vector of prediction scores \hat{\mathbf{p}}=[\hat{p}_{1},\dots,\hat{p}_{K}] over the K classes, where each \hat{p}_{k} is often interpreted as the probability that the sample belongs to class k . Each ground-truth label is encoded as a one-hot vector \mathbf{y}\in\{0,1\}^{K} with a single 1

Read the complete text