Enforcing Calibration in Multi-Output Probabilistic Regression with Pre-rank Regularization

Naomi Desobry, Elnura Zhalieva, Souhaib Ben Taieb · 2025

Read Enforcing Calibration in Multi-Output Probabilistic Regression with Pre-rank Regularization by Naomi Desobry, Elnura Zhalieva, Souhaib Ben Taieb (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21273.txt.

Introduction

Naomi Desobry, Elnura Zhalieva, and Souhaib Ben Taieb address calibration in forecasts that predict several outcomes at once. A calibrated model is right about its own uncertainty, but checking and fixing that property for many outputs at once is much harder than for a single output.

Older tools only diagnose problems, using summary scores called pre-ranks that compress each forecast and outcome pair into a single number. The authors turn diagnosis into training: a penalty term pushes those summaries toward the pattern expected from a calibrated model. Their penalty covers both single outputs and joint behavior, and a new summary based on principal components tracks directions of greatest spread. Across eighteen real data sets, the penalty improves calibration without hurting accuracy.

Catalog record

CREATORNaomi Desobry, Elnura Zhalieva, Souhaib Ben Taieb
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21273

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enforcing Calibration in Multi-Output Probabilistic Regression with Pre-rank Regularization Naomi Desobry, Elnura Zhalieva, Souhaib Ben Taieb Abstract Probabilistic models must be well calibrated to support reliable decision-making. While calibration in single-output regression is well studied, defining and achieving multivariate calibration in multi-output regression remains considerably more challenging. The existing literature on multivariate calibration primarily focuses on diagnostic tools based on pre-rank functions, which are projections that reduce multivariate prediction-observation pairs to univariate summaries to detect specific types of miscalibration. In this work, we go beyond diagnostics and introduce a general regularization framework to enforce multivariate calibration during training for arbitrary pre-rank functions. This framework encompasses existing approaches such as highest density region calibration and copula calibration. Our method enforces calibration by penalizing deviations of the projected probability integral transforms (PITs) from the uniform distribution, and can be added as a regularization term to the loss function of any probabilistic predictor. Specifically, we propose a regularization loss that jointly enforces both marginal and multivariate pre-rank calibration. We also introduce a new PCA-based pre-rank that captures calibration along directions of maximal variance in the predictive distribution, while also enabling dimensionality reduction. Across 18 real-world multi-output regression datasets, we show that unregularized models are consistently miscalibrated, and that our methods significantly improve calibration across all pre-rank functions without sacrificing predictive accuracy. Introduction Probabilistic models output full predictive distributions rather than point estimates, enabling principled uncertainty-aware decision-making in domains such as meteorology, finance, and medical diagnosis ( Murphy and Winkler 1984 ; Krzysztofowicz and Evans 2008 ; de Lima Silva et al. 2020 ; Önkal and Muradoǧlu 1994 ; Gulshan et al. 2016 ; Guizilini et al. 2019 ) . However, to be reliable, these predictions must be calibrated , that is, their predicted probabilities must align with the observed frequencies. In single-output regression, calibration is well understood and can be evaluated using tools such as the Probability Integral Transform (PIT). Deviations from perfect calibration, referred to as miscalibration, can be corrected either during training via regularization techniques based on scoring rule decompositions ( Wilks 2018 ; Wessel et al. 2025 ) , or post hoc using recalibration methods such as isotonic regression ( Kuleshov, Fenner, and Ermon 2018 ) or kernel-based adjustments ( Dheur and Taieb 2023 ) . Throughout this work, we refer to calibration in single-output regression as univariate calibration. Multivariate calibration , by contrast, concerns the calibration of a multivariate target and is considerably more difficult to evaluate and achieve. When producing probabilistic predictions for such targets, correctly specifying the marginal distributions is not sufficient; the predictions must also accurately capture the dependencies and joint structure across target dimensions. Although several tools have been proposed to assess specific aspects of multivariate calibration ( Chung, Char, and Schneider 2024 ; Ziegel and Gneiting 2013 ) , defining general-purpose, interpretable, and effective calibration methods for the multivariate setting remains an open challenge. One approach to evaluating multivariate calibration involves the use of pre-rank functions, which are scalar summaries of prediction-observation pairs that extend univariate rank-based diagnostics to the multivariate setting ( Allen, Ziegel, and Ginsbourger 2023 ) . Each pre-rank targets a specific aspect of miscalibration, such as marginal calibration or discrepancies in summary statistics like location, scale, or dependence structure. By projecting complex multivariate predictions onto interpretable scalar quantities, pre-rank functions provide a flexible and general framework for assessing different dimensions of probabilistic calibration. In this work, we go beyond diagnostic tools and propose a method to directly enforce multivariate calibration by incorporating a regularization term into the training loss. This term penalizes miscalibration with respect to a collection of pre-rank functions. We further introduce a novel pre-rank based on Principal Component Analysis (PCA), which projects prediction-observation pairs onto directions of maximal variance in the predictive distribution, thereby capturing calibration along statistically meaningful directions. Additionally, we propose a regularization loss that jointly enforces both marginal and multivariate pre-rank calibration. When combined with the PCA-based pre-rank, our approach also enables dimensionality reduction and improves computational efficiency. Empirically, our method consistently improves calibration across all pre-rank functions without compromising predictive accuracy. We make the following main contributions: • We conduct a large-scale empirical study on 18 real-world multi-output regression datasets to evaluate the probabilistic calibration of unregularized models across a diverse set of pre-rank functions. • We propose a general regularization framework that can be integrated into the training of any probabilistic predictor to enforce multivariate calibration with respect to user-specified pre-rank functions. Our approach also includes a joint regularization loss that enforces both marginal and multivariate calibration. When combined with our PCA-based pre-rank, the method detects calibration along the top principal components of the predictive covariance while also serving as a dimensionality reduction technique. • We validate our framework on 18 benchmark datasets and show that it consistently improves calibration across all pre-rank metrics without compromising predictive accuracy. Background We consider a multivariate regression setting where inputs X\in\mathcal{X}\subseteq\mathbb{R}^{L} and targets Y\in\mathcal{Y}\subseteq\mathbb{R}^{D} are jointly distributed. The target Y=(Y_{1},\dots,Y_{D}) has dimension D\geq 1 . Our goal is to estimate the true conditional distribution F_{Y|X} from a finite dataset \mathcal{D}=\{(X_{i},Y_{i})\}_{i=1}^{N} . To this end, we define a probabilistic predictor F_{\theta}:\mathcal{X}\rightarrow\mathcal{F} , where \mathcal{F}\subseteq\mathcal{P}(\mathbb{R}^{D}) is a class of admissible probability distributions over \mathbb{R}^{D} , \mathcal{P}(\mathbb{R}^{D}) denotes the space of all probability distributions on \mathbb{R}^{D} , and \theta represents the model parameters. For any input x\in\mathcal{X} , the model outputs a predictive cumulative distribution function \hat{F}_{Y|X=x}\in\mathcal{F} , with corresponding density \hat{f}_{Y|X=x} . This distribution may be available in closed form (e.g., a multivariate Gaussian) or approximated via samples. We learn the model parameters \theta by minimizing a proper scoring rule over a training dataset, thereby encouraging the predictive distribution F_{\theta}(x) to align with the true conditional distribution of Y given X=x . A scoring rule S:\mathcal{F}\times\mathcal{Y}\to\mathbb{R} assigns a numerical score to each predictive distribution \hat{F}\in\mathcal{F} and observed outcome y\in\mathcal{Y} . It is called proper if it is minimized in expectation when \hat{F} equals the true distribution, and strictly proper if the minimizer is unique. Two widely used examples are the negative log-likelihood (NLL), \mathrm{NLL}(\hat{F},y)=-\log\hat{f}(y) , where \hat{f} is the density of \hat{F} , and the energy score (ES), \mathrm{ES}(\hat{F},y)=\mathbb{E}_{\hat{Y}\sim\hat{F}}\|\hat{Y}-y\|-\frac{1}{2}\mathbb{E}_{\hat{Y},\hat{Y}^{\prime}\sim\hat{F}}\|\hat{Y}-\hat{Y}^{\prime}\|. This estimation strategy, known as optimum score estimation ( Gneiting and Raftery 2007 ) , allows flexible learning of predictive distributions. However, model misspecification and limited data may lead to biased or miscalibrated models, and the choice of scoring rule can also affect the accuracy, robustness, and calibration of the resulting model. Univariate calibration. To better understand calibration in the multivariate setting, we briefly recall probabilistic calibration in the univariate setting. Let X\in\mathcal{X} and Y\in\mathbb{R} be random variables with conditional distribution F_{Y|X} , and let \hat{F}_{Y|X} be a probabilistic predictor. Definition 1 . \hat{F}_{Y|X} is said to be PIT-calibrated if the probability integral transform (PIT), Z=\hat{F}_{Y|X}(Y), is uniformly distributed on [0,1] , that is, F_{Z}(\alpha)=\alpha\quad\text{for all }\alpha\in[0,1]. (1) This property guarantees that the predicted distribution is statistically consistent with the observed outcomes. This condition holds if \hat{F}_{Y|X} matches the true conditional CDF. In practice, the deviation of the PIT distribution from uniformity can be quantified using the probabilistic calibration error (PCE), defined as \text{PCE}(F_{\theta},\mathcal{D})=\frac{1}{M}\sum_{j=1}^{M}\left|\alpha_{j}-\hat{F}_{Z}(\alpha_{j})\right|, (2) where \{\alpha_{j}\}_{j=1}^{M} is a grid of quantile levels such that \alpha_{j}\in[0,1] , and

Read the complete text