Probabilistic Consistency in Machine Learning and Its Connection to Uncertainty Quantification
Paul N. Patrone, Anthony J. Kearsley · 2025
Read Probabilistic Consistency in Machine Learning and Its Connection to Uncertainty Quantification by Paul N. Patrone, Anthony J. Kearsley (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21670.txt.
Introduction
Paul Patrone and Anthony Kearsley ask what a machine learning classifier's scores really mean. They start from a question doctors know well: what share of this group is truly positive? That share is called prevalence. The word sounds plain, yet it hides more than one meaning, and each meaning rests on hidden premises about the model.
From the best possible classifiers in the Bayes sense, they build a level-set theory. It shows that some self-consistent models are exactly the same as class-conditional probability distributions. That match turns vague confidence into uncertainty that can be measured. For medicine and measurement work, such numbers can steer care or policy. Knowing when a black box is secretly a probability distribution, and when it is not, is basic hygiene for the field.
Catalog record
| CREATOR | Paul N. Patrone, Anthony J. Kearsley |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-21670
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Probabilistic Consistency in Machine Learning and Its Connection to Uncertainty Quantification Paul N. Patrone, Anthony J. Kearsley Abstract Machine learning (ML) is often viewed as a powerful data analysis tool that is easy to learn because of its black-box nature. Yet this very nature also makes it difficult to quantify confidence in predictions extracted from ML models, and more fundamentally, to understand how such models are mathematical abstractions of training data. The goal of this paper is to unravel these issues and their connections to uncertainty quantification (UQ) by pursuing a line of reasoning motivated by diagnostics. In such settings, prevalence – i.e. the fraction of elements in class – is often of inherent interest. Here we analyze the many interpretations of prevalence to derive a level-set theory of classification , which shows that certain types of self-consistent ML models are equivalent to class-conditional probability distributions. We begin by studying the properties of binary Bayes optimal classifiers, recognizing that their boundary sets can be reinterpreted as level-sets of pairwise density ratios. By parameterizing Bayes classifiers in terms of the prevalence, we then show that they satisfy important monotonicity and class-switching properties that can be used to deduce the density ratios without direct access to the boundary sets. Moreover, this information is sufficient for tasks such as constructing the multiclass Bayes-optimal classifier and estimating inherent uncertainty in the class assignments. In the multiclass case, we use these results to deduce normalization and self-consistency conditions, the latter being equivalent to the law of total probability for classifiers. We also show that these are necessary conditions for arbitrary ML models to have valid probabilistic interpretations. Throughout we demonstrate how this analysis informs the broader task of UQ for ML via an uncertainty propagation framework. 1 1 footnotetext: National Institute of Standards and Technology, Gaithersburg MD 20899, USA 2 2 footnotetext: Department of Medicine, Division of Infectious Diseases and Immunology, University of Massachusetts Chan Medical School, Worcester, MA, 01655, USA 1 Introduction From our perspective, the formulation of a complete uncertainty quantification (UQ) framework for machine learning (ML) remains an unresolved problem. Several works have addressed empirical aspects of this problem through the lens of Monte Carlo methods and related sampling techniques [ 1 , 2 , 3 , 4 ] . Moreover, it is well known that classification theory in general can be grounded in probability [ 5 ] . Indeed, this perspective has led to important observations, such as the recognition that the Bayes classifier, which minimizes the expected classification error, also arises from optimization of surrogate loss functions [ 6 , 7 , 8 , 9 ] . In this context, many authors have also addressed elements of uncertainty estimation as a task in calibration , i.e. ensuring that probabilistic ML outputs are consistent with distributional information about data [ 10 , 11 , 12 , 13 , 14 , 15 , 16 ] . However, these works have not considered more fundamental and perhaps subtle questions: (i) are ML predictions always mathematically consistent with probability theory; (ii) can one meaningfully associate aleatoric uncertainty estimates with arbitrary ML models [ 17 ] ; and thus, (iii) do they even permit meaningful UQ? Stated differently, when is calibration possible, and what are the effects that would inform the calibration process? A key difficulty in answering these questions arises from the fact that it is not always clear how arbitrary ML models are abstract representations of training data. For example, the Bayes classifier is known to be a manifestation of underlying conditional distributions \Pr[r|C] of “inputs” r generated by class C . But by the same token, it is not obvious that such classifiers contain all of the information about these distributions. Thus, it is possible that certain ML models can provide minimum error guarantees in a global sense, * * * That is, by minimizing the expected classification error. but not pointwise estimates of their confidence. More generally, if an arbitrary classifier is to admit meaningful UQ, then we assert that it must be equivalent in some sense to the underlying conditional distributions \Pr[r|C] . The purpose of this manuscript is to unravel such issues by deriving a level-set theory of classification that establishes necessary conditions for there to be a mathematical correspondence between classifiers \hat{C} and the distributions \Pr[r|C] . Importantly, this correspondence permits a deeper analysis of UQ for ML in terms of a formal uncertainty propagation framework and thereby facilitates tasks such as estimation of inherent uncertainty (a variation on aleatoric uncertainty) in the class labels. The main idea of this analysis is to study the properties of binary Bayes classifiers, and in particular their boundary sets, which we reinterpret as level sets of pairwise density ratios quantifying the likelihood that an input r arises from a given class. By parameterizing these level sets in terms of the prevalence (fraction of elements in a class), we arrive at an important montonicity property that holds in general for Bayes classifiers. Without having explicit access to the boundary sets, we then use this property to deduce the density ratios, and hence information about \Pr[r|C] , in terms of the prevalence value at which the class changes. In extending this result to the multiclass setting, we also identify normalization and self-consistency criteria, the latter expressing the law of total probability for classifiers. By recourse to the density ratios, our analysis then: (i) demonstrates the sense in which monotonicity, self-consistency, and normalizability are necessary conditions for establishing the correspondence between \hat{C} and \Pr[r|C] ; and (ii) shows how the aforementioned analyses can account for distribution shifts due to changes in prevalence. Because probability is central to our analysis, a related goal of this manuscript is to unravel the role of uncertainty quantification (UQ) as a tool for understanding ML more generally, not just assessing confidence in its predictions. In other words, we take the position that UQ is inherent to ML, and to analyze ML is to study the theory of UQ for classification. To realize this in practice, we adopt a small but important change in perspective. Canonical approaches to ML often directly consider the conditional probability \Pr[C|r] of class C conditioned on the input r [ 18 , 1 , 2 , 3 , 4 , 19 , 20 ] . But it is well known that \displaystyle\Pr[C|r]=\frac{\Pr[r|C]\Pr[C]}{\Pr[r]}=\frac{\Pr[r|C]\Pr[C]}{\sum_{C^{\prime}}\Pr[r|C^{\prime}]\Pr[C^{\prime}]}, (1) where \Pr[C] is the prevalence of class C . Equation ( 1 ) is sometimes used to motivate the definition of a generative classifier as one that directly models \Pr[r|C] [ 21 ] . But Eq. ( 1 ) demonstrates that this distinction is superficial, as it simply amounts to whether \Pr[r|C] is implicit or explicit in the modeling. Moreover, it is well known that one can objectively estimate \Pr[C] without recourse to classification or Bayesian priors per se [ 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 ] . In this context, we argue that the study of ML is more natural when reformulated in terms of \Pr[r|C] , since it isolates and makes explicit the way in which classifiers represent data. This shift in perspective is central to our analysis, and