Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score

Stefan Haas, Luca Killmaier, Alireza Javanmardi, Eyke Hüllermeier · 2026

Read Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score by Stefan Haas, Luca Killmaier, Alireza Javanmardi, Eyke Hüllermeier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24959.txt.

Introduction

Stefan Haas, Luca Killmaier, Alireza Javanmardi, and Eyke Hüllermeier build a conformal prediction method for ordinal classification. Ordinal problems, where labels have a natural order, arise in medicine and finance, and mistakes differ in severity. The approach gives distribution free sets with coverage guarantees. The authors propose the ranked probability score, a proper scoring rule over cumulative forecasts, as the nonconformity function. The score reflects ordinal risk naturally, yet conformal work has neglected it.

Used as the nonconformity measure, the score yields median centred contiguous prediction sets. The method is model agnostic and handles assessed and grouped ordered outcomes. It also runs faster than greedy interval selection procedures. Across image and tabular datasets, the sets stay in one piece. The balance between set width and the size of ordinal miscoverage beats conformal methods. Haas and colleagues argue the score deserves to become the default choice.

Catalog record

CREATORStefan Haas, Luca Killmaier, Alireza Javanmardi, Eyke Hüllermeier
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24959

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score Stefan Haas, Luca Killmaier, Alireza Javanmardi, Eyke Hüllermeier Abstract Ordinal classification (OC) arises in high-stakes domains such as medicine and finance, where uncertainty quantification must account for the severity of ordinal errors. Conformal prediction (CP) provides distribution-free prediction sets with marginal coverage guarantees; however, its practical effectiveness depends critically on the choice of nonconformity function. We introduce a CP method for ordinal classification based on the ranked probability score (RPS), a proper scoring rule defined over cumulative predictive distributions. Although it reflects ordinal risk quite naturally, it has largely been neglected in conformal ordinal prediction (COP). When used as a measure of nonconformity, RPS yields median-centered contiguous prediction sets by construction. The method is model-agnostic, supports both assessed and grouped ordered categorical outcomes, and permits efficient implementation compared to greedy interval selection procedures. Across multiple ordinal image and tabular datasets, RPS-based CP produces contiguous prediction sets and strikes a favorable balance between prediction set width and the magnitude of ordinal miscoverage relative to existing CP methods. 1 Introduction Ordinal classification (OC), also known as ordinal regression in statistics 42 , refers to classification problems in which class labels exhibit a natural linear order. Representative applications include medical diagnosis 2 , age estimation 12 , and credit risk assessment 32 . Despite its prevalence in high-stakes domains, most work in OC has primarily focused on improving point prediction performance 61 , 46 , 53 , whereas uncertainty quantification (UQ) has attracted attention only recently 30 , 29 . Here, uncertainty for a query \boldsymbol{x}_{q} is typically represented by the conditional predictive distribution over ordered labels, p(y\mid\boldsymbol{x}_{q}) , produced by a probabilistic predictor. Alternatively, uncertainty can be represented by set-valued predictions. Conformal prediction (CP) offers a principled, model-agnostic framework for post-hoc construction of prediction sets 60 , 68 , 70 , 5 . It calibrates an underlying (heuristic) uncertainty estimate to achieve finite-sample, distribution-free marginal coverage at a user-specified miscoverage rate \alpha . Instead of committing to a single label y\in\mathcal{Y} , CP outputs a set \mathcal{C}_{\alpha}(\boldsymbol{x}_{q})\subseteq\mathcal{Y} of plausible labels, whose size reflects predictive uncertainty at the query \boldsymbol{x}_{q} . However, producing sets in conformal ordinal prediction (COP) that are both informative and consistent with the ordinal structure of the label space remains challenging. A key requirement for COP is that prediction sets be contiguous 41 , 71 , 21 . For example, consider age estimation 61 , 73 from images, where a latent continuous variable (e.g., age) is discretized into ordered categories, with \mathcal{Y}=\{\texttt{baby},\texttt{child},\texttt{teenager},\texttt{adult},\texttt{senior}\} . A valid prediction set should only contain adjacent age categories, e.g., \mathcal{C}_{\alpha}(\boldsymbol{x}_{q})=\{\texttt{child},\texttt{teenager},\texttt{adult}\} , whereas non-contiguous sets such as \mathcal{C}_{\alpha}(\boldsymbol{x}_{q})=\{\texttt{child},\texttt{senior}\} may appear unreasonable. Figure 1: (Left) Illustration of an assessed ordered categorical variable (survival prognosis for melanoma 47 ) with extreme disagreement between physicians. To faithfully quantify uncertainty, the prediction set must be contiguous and include all intermediate classes between the conflicting assessments. (Right) Illustration of a grouped ordered categorical variable (age estimation), where unimodal predictive modeling is well justified and naturally leads to contiguous prediction sets 39 , 48 . Another important category, alongside the previously described grouped ordered categorical variables, is the class of assessed ordered categorical variables 3 , in which human experts assign labels, as in financial risk assessment or medical survival prognosis. In these contexts, errors tend to be inherently larger due to inter-expert disagreement, which makes maintaining contiguity even more crucial for accurately capturing uncertainty. For example, consider physician opinions regarding survival prognosis for stage IV melanoma, which may be polarized, resulting in large clusters at <1 year (very pessimistic group) and \gg 5 years (optimistic group influenced by immunotherapy outcomes). To properly quantify uncertainty in such cases, the prediction set should not be limited to the two most frequent categories, i.e., \mathcal{C}_{\alpha}(\boldsymbol{x}_{q})=\{\texttt{<1 year},\texttt{$\gg$5 years}\} ; instead, it should encompass the entire range of plausible categories, \mathcal{C}_{\alpha}(\boldsymbol{x}_{q})=\{\texttt{<1 year},\texttt{1-3 years},\texttt{3-5 years},\texttt{>5 years},\\ \texttt{$\gg$5 years}\} . Otherwise, uncertainty measured by the size of the prediction set will be severely underestimated (see Figure 1 ). Another important aspect of COP that has received limited attention is the severity of miscoverage when the true label lies outside the prediction set. For instance, in financial risk assessment, if the true label is y=\texttt{very high} while \mathcal{C}_{\alpha}(x)=\{\texttt{low},\texttt{moderate}\} , the resulting miscoverage is substantial and could lead to catastrophic risk misestimation. Ideally, when coverage fails, the true label should lie as close as possible to the boundary of the prediction set, minimizing the impact of miscoverage. Under this criterion, a larger set such as \mathcal{C}_{\alpha}(x)=\{\texttt{low},\texttt{moderate},\texttt{high}\} may be preferable to a smaller set \mathcal{C}_{\alpha}(x)=\{\texttt{low},\texttt{moderate}\} , even though both fail to cover the true label and the larger set is less efficient in the classical CP sense. This observation motivates uncertainty quantification methods for OC that account not only for coverage and efficiency, but also for the ordinal distance incurred under miscoverage. These challenges motivate a model-agnostic conformal approach to OC that can be combined with arbitrary loss functions, accommodates both grouped and assessed ordered targets, and produces meaningful contiguous prediction sets that accurately quantify uncertainty. Moreover, such a method should leverage the entire predictive probability distribution in an unbiased and computationally efficient manner when constructing prediction sets. These requirements are not met by existing approaches (cf. Section 2 ). In this paper, we advocate the ranked probability score (RPS) 24 as a nonconformity measure for conformal prediction in OC. RPS is a proper scoring rule 27 for ordinal outcomes that incentivizes truthful probability estimation and explicitly accounts for the linear structure of the label space. Despite being well-established in the forecasting literature 45 , 43 , it has only recently been recognized as a theoretically grounded metric for evaluating probabilistic ordinal classifiers in machine learning 25 . To the best of our knowledge, RPS has not yet been proposed as a nonconformity measure for CP in OC. The main contributions of this paper are as follows: • We propose RPS as a proper, model-agnostic nonconformity measure for CP in OC. • We provide theoretical guarantees for desirable properties in OC: RPS-based conformal prediction sets (i) satisfy marginal coverage , (ii) are nested with respect to the miscoverage level ( \alpha ), and (iii) are contiguous . • Furthermore, we show that RPS-based prediction sets directly optimize ordinal risk under oracle conditional coverage , measured as set-based l_{1} error, in contrast to mode-centered approaches which primarily target set efficiency. • We show that RPS-based conformal prediction is computationally efficient, scaling linearly with both the number of labels and the number of calibration points. • Finally, we empirically validate our approach on ordinal image and tabular datasets, showing that median-centered RPS-based prediction sets strike a favorable balance between interval width and ordinal miscoverage magnitude. 2 Related Work Ordinal Classification addresses the problem of predicting discrete ordered labels as commonly encountered in many high-stakes domains, including medicine 23 , 54 , 63 and finance 32 . Unlike multinomial classification, where class labels are unordered, OC must account for the inherent order among classes, implying that misclassification costs typically increase with an increasing gap between predicted label \hat{y} and true label y . At the same time, OC differs from regression in

Read the complete text