The relation of bias with risk in empirically constrained inferences
Dalton A R Sakthivadivel · 2025
Read The relation of bias with risk in empirically constrained inferences by Dalton A R Sakthivadivel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03699.txt.
Introduction
Dalton Sakthivadivel links two classic ideas in statistical inference: maximum entropy and Bayes-optimal classification. The 2025 paper shows that choosing the maximum entropy probability assignment acts as a universally Bayes optimal decision rule. That holds when what one knows about the data is stated as limits on expected loss. The work then extends the result to cases where those expected loss observations are themselves uncertain. A generalized form of Sanov's theorem carries the argument. The paper is a compact theory piece for readers in inference and machine learning.
Catalog record
| CREATOR | Dalton A R Sakthivadivel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-03699
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The relation of bias with risk in empirically constrained inferences Dalton A R Sakthivadivel Abstract . We give some results relating asymptotic characterisations of maximum entropy probability measures to characterisations of Bayes optimal classifiers. Our main theorems show that maximum entropy is a universally Bayes optimal decision rule given constraints on one’s knowledge about some observed data in terms of an expected loss. We will extend this result to the case of uncertainty in the observations of expected losses by generalising Sanov’s theorem to distributions of constraint values. 2020 Mathematics Subject Classification 60F10, 62B10, 62C10, 82M60 1. Introduction Given the probability space (\Omega,\mathscr{F},\mu) , \Omega a Polish space, let X\sim P\coloneqq\mu_{*} be a random variable X\colon\Omega\to M , and Y\colon\Pi\to N on (\Pi,\mathscr{G},\nu) be a label associated to X by some unknown \mu_{*} -measurable mapping f\colon X\mapsto Y . One is often interested in approximating f by some hypothetical parametric function h_{\theta}\colon X\mapsto Y , classifying observed values of X according to their ‘best possible’ labels. Given a loss function \mathcal{L}(h_{\theta}(x),y) , the risk of misclassification or generalisation error is the expected error on the joint measure space R(h_{\theta})=\int_{N}\int_{M}\mathcal{L}(h_{\theta}(x),y)p(x,y)\differential x\differential y. Under obvious lower semi-continuity assumptions on the risk there is a unique classifier g of least risk such that for any h_{\theta} , R(h_{\theta})\geqslant R(g). We call g the Bayes classifier. It can be shown that the output of the Bayes classifier is determined by minimising the conditionally expected error, (1.1) g(x)=\argmin_{z}\int_{N}\mathcal{L}(z,y)p(y\mid x)\differential y. In this case, one often says g is Bayes optimal with respect to the prior probability of y , using the Bayesian inversion p(x\mid y)\pi(y) . For standard references see [ Ber85 , Mac03 , SSBD14 ] . We claim one can turn this story around and see it as the inference of a posterior consistent with our knowledge of (i) the value of X and (ii) the loss function on possible classes associated to X . In other words, from the existence of a Bayes classifier follows an optimal ‘guess’ of the distribution over Y given the information we have about X , such that the loss function and the distance of h_{\theta} from g constrains the form of P(Y\mid X) . The results here are motivated by this claim and the accompanying question: how should that inference be performed—what is the optimal such guess? Moreover, what determines optimality in this context? The principle of maximum entropy—in its asymptotic form—has obvious relevance, envisioned by e.g. Skyrms as a method for supposition which is compatible with Bayesian model evidence evaluation [ Sky87 ] and by Shore–Johnson as an inference procedure consistent with some given knowledge but including no extraneous assumptions [ SJ80 ] . We will presently describe how these properties are relevant to the matter at hand. The concept behind ( 1.1 ) is that for a given X=x we want to predict a z and compare it against any given Y=y using \mathcal{L} , then consider the total result across all y ’s possibly associated to x , weighted by their likelihoods. Note that this lattermost property of ( 1.1 ) can be thought of as controlling the risk of overfitting the classifier to a rare observation of Y given X . The classifier which minimises this expected loss is the Bayes classifier. An example is that of a quadratic loss function \mathcal{L}(h_{\theta}(x),y)=(h_{\theta}(x)-y)^{2} , in which case the best value for h_{\theta}(x)=z can be computed as \E[Y\mid X] . By pure reasoning, one knows this is equal to the value of Y observed on average given X , so minimises the conditionally expected difference between classifier outputs and values of Y , which is ( 1.1 ). The computation follows precisely this logic by setting the derivative of ( 1.1 ) with respect to z , 2\E[z-y\mid X] , equal to zero, and noting this is equivalent to z-\E[Y\mid X]=0 by linearity and independence of the variable z from y . Suppose we make a random choice of z weighted according to \mathcal{L} and assume the image of the classifier is uniquely determined by \theta . Obviously the distribution of classifier outputs is not the true distribution of labels, since y is fixed. Conversely , however, if we fix a classifier and can measure its error empirically, then we may ask for the most probable distribution of possible Y values given (i) a value X=x and (ii) our knowledge that h_{\theta}(x) has some conditionally expected error \E[\mathcal{L}(h_{\theta}(x),y)\mid x]=\xi . Furthermore, if we know the minimal error \xi^{*} , we can then compare that distribution to the distribution we would obtain if h_{\theta}(x) were the Bayes classifier so that \xi=\xi^{*} , using the relative entropy or the distance between \xi and \xi^{*} to estimate how far h_{\theta} is from g (indeed these should be the same procedures in some suitable sense). That is to say: on one hand, we can determine what the true distribution most likely is if we know \xi . On the other, we can ask what the statistics of the distribution would need to be for a fixed h_{\theta}(x) to be its Bayes classifier. Together, we can determine the minimum possible divergence between the true posterior and any inferred posterior given h_{\theta}(x) , and how that divergence varies with respect to changing \theta . From this we claim finding the Bayes optimal classifier empirically involves inferring the true posterior and then studying large deviations in P_{\theta}(Y\mid X) constrained by an empirically expected error \xi . By Sanov’s theorem, the probability of deviations in empirical measures from a true measure are dominated by their closest points to that true measure, measured by the relative entropy. Likewise it is known that under certain desiderata on an inference (outlined in [ SJ80 ] and reviewed in the following section) the entropy -\int P\log P-\lambda\left(\int VP-C\right) is singled out as the functional to be extremised, where given a reference measure Q , V is any measurable function with growth such that e^{-\lambda_{C}V} is locally L^{1} with respect to Q , for some value of \lambda determined by C . We will here discuss a collection of results linking the optimality of the Bayes classifier with the optimality of the principle of maximum entropy as a form of constrained inference. In particular, our proofs will use the theory of large deviations to link constrained probabilities with maximum a posteriori estimates. The generality of our framework goes well beyond quadratic utility functions, addressing e.g. a critique of Berger concerning the use of mean squared error. On the other hand, we will only consider unimodal distributions; that is, we will assume \mathcal{L} is lower semi-continuous, convex, and at least twice continuously differentiable. We will assume the extremal point of the relative entropy is also unique. Necessary and sufficient conditions for this assumption to hold are given in [ Csi84 ] . We will finally implicitly assume absolute continuity wherever necessary, e.g. in discussions of the relative entropy. We will also assume the true measure and true labels are known, say, for the purpose of training or fitting parametric distributions to the true measure. We are grateful to K Dill and K Friston for many useful discussions around