Variations on the Expectation due to Changes in the Probability Measure Thanks: This work is supported in part by the European Commission through the H2020-MSCA-RISE-2019 project 872172; the French National Agency for Research (ANR) through the Project ANR-21-CE25-0013 and the project ANR-22-PEFT-0010 of the France 2030 program PEPR Réseaux du Futur; and in part by the Agence de l’innovation de défense (AID) through the project UK-FR 2024352. Samir M. Perlaza 1 2 3 and Gaetan Bisson 3 Abstract In this paper, closed-form expressions are presented for the variation of the expectation of a given function due to changes in the probability measure used for the expectation. They unveil interesting connections with Gibbs probability measures, mutual information, and lautum information. I Introduction Let m be a positive integer and denote by \triangle(\mathbb{R}^{m}) the set of all probability measures on the measurable space \left(\mathbb{R}^{m},\mathscr{B}\left(\mathbb{R}^{m}\right)\right) , with \mathscr{B}\left(\mathbb{R}^{m}\right) being the Borel \sigma -algebra on \mathbb{R}^{m} . Given a Borel measurable function h:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R} , consider the functional \mathsf{G}_{h}:\mathbb{R}^{n}\times\triangle(\mathbb{R}^{m})\times\triangle(\mathbb{R}^{m})\to\mathbb{R} such that \displaystyle\mathsf{G}_{h}\left(x,P_{1},P_{2}\right) \displaystyle= \displaystyle\int h(x,y)\mathrm{d}P_{1}(y)-\int h(x,y)\mathrm{d}P_{2}(y), (1) which quantifies the variation of the expectation of the measurable function h due to changing the probability measure from P_{2} to P_{1} . Such a functional is defined when both integrals exist and are finite. In order to define the expectation of \mathsf{G}_{h}\left(x,P_{1},P_{2}\right) with respect to x , the structure formalized below is required. Definition 1 . A family P_{Y|X}\triangleq(P_{Y|X=x})_{x\in\mathbb{R}^{n}} of elements of \triangle(\mathbb{R}^{m}) indexed by \mathbb{R}^{n} is said to be a conditional probability measure if, for all sets \mathcal{A}\in\mathscr{B}\left(\mathbb{R}^{m}\right) , the map \displaystyle\left\{\begin{array}[]{@{}r@{~}c@{~}l@{}}\mathbb{R}^{n}\penalty\ &\longrightarrow\hfil\penalty\ &[0,1]\\ x\penalty\ &\longmapsto\hfil\penalty\ &P_{Y|X=x}(\mathcal{A})\end{array}\right. is Borel measurable. The set of all such conditional probability measures is denoted by \triangle\left(\mathbb{R}^{m}|\mathbb{R}^{n}\right) . In this setting, consider the functional \bar{\mathsf{G}}_{h}:\triangle\left(\mathbb{R}^{m}|\mathbb{R}^{n}\right)\times\triangle\left(\mathbb{R}^{m}|\mathbb{R}^{n}\right)\times\triangle\left(\mathbb{R}^{n}\right)\to\mathbb{R} such that \displaystyle\bar{\mathsf{G}}_{h}\left(P_{Y|X}^{(1)},P_{Y|X}^{(2)},P_{X}\right) \displaystyle= \displaystyle\int\mathsf{G}_{h}\left(x,P_{Y|X=x}^{(1)},P_{Y|X=x}^{(2)}\right)\mathrm{d}P_{X}(x).\quad (5) This quantity can be interpreted as the variation of the integral (expectation) of the function h when the probability measure changes from the joint probability measure P_{Y|X}^{(1)}P_{X} to another joint probability measure P_{Y|X}^{(2)}P_{X} , both in \triangle\left(\mathbb{R}^{m}\times\mathbb{R}^{n}\right) . This follows from ( 5 ) by observing that \displaystyle\bar{\mathsf{G}}_{h}\left(P_{Y|X}^{(1)},P_{Y|X}^{(2)},P_{X}\right) (6) \displaystyle= \displaystyle\int h(x,y)\mathrm{d}P_{Y|X}^{(1)}P_{X}(y,x)-\int h(x,y)\mathrm{d}P_{Y|X}^{(2)}P_{X}(y,x). Special attention is given to the quantity \bar{\mathsf{G}}_{h}\left(P_{Y},P_{Y|X},P_{X}\right) , for some P_{Y|X}\in\triangle\left(\mathbb{R}^{m}|\mathbb{R}^{n}\right) , with P_{Y} being the marginal of the joint probability measure P_{Y|X}\cdot P_{X} . That is, for all sets \mathcal{A}\in\mathscr{B}\left(\mathbb{R}^{m}\right) , \displaystyle P_{Y}\left(\mathcal{A}\right) \displaystyle= \displaystyle\int P_{Y|X=x}\left(\mathcal{A}\right)\mathrm{d}P_{X}(x). (7) Its relevance stems from the fact that it captures the variation of the expectation of the function h when the probability measure changes from the joint probability measure P_{Y|X}P_{X} to the product of its marginals P_{Y}P_{X} . That is, \displaystyle\bar{\mathsf{G}}_{h}\left(P_{Y},P_{Y|X},P_{X}\right) (8) \displaystyle= \displaystyle\int\left(\int h(x,y)\mathrm{d}P_{Y}(y)-\int h(x,y)\mathrm{d}P_{Y|X=x}(y)\right)\mathrm{d}P_{X}(x) \displaystyle= \displaystyle\int h(x,y)\mathrm{d}P_{Y}P_{X}(y,x)-\int h(x,y)\mathrm{d}P_{Y|X}P_{X}(y,x). I-A Novelty and Contributions This work makes two key contributions: First, it provides a closed-form expression for the variation \mathsf{G}_{h}\left(x,P_{1},P_{2}\right) in ( 1 ) for a fixed x\in\mathbb{R}^{n} and two arbitrary probability measures P_{1} and P_{2} , formulated explicitly in terms of information-theoretic quantities. Second, it derives a closed-form expression for the expected variation \bar{\mathsf{G}}_{h}\left(P_{Y|X}^{(1)},P_{Y|X}^{(2)},P_{X}\right) in ( 5 ), again in terms of information measures, for arbitrary conditional probability measures P_{Y|X}^{(1)} , P_{Y|X}^{(2)} , and marginal measure P_{X} . A further contribution of this work is the derivation of specific closed-form expressions for \bar{\mathsf{G}}_{h}\left(P_{Y},P_{Y|X},P_{X}\right) in ( 8 ), which reveal deep connections with both mutual information [ 1 , 2 ] and lautum information [ 3 ] . Notably, when P_{Y|X} is a Gibbs conditional probability measure, this variation simplifies (up to a constant factor) to the sum of the mutual and lautum information induced by the joint distribution P_{Y|X}P_{X} . Although these results were originally discovered in the analysis of generalization error of machine learning algorithms, see for instance [ 4 , 5 , 6 , 7 , 8 ] , where the function h in ( 1 ) was assumed to represent an empirical risk, this paper presents such results in a comprehensive and general setting that is no longer tied to such assumptions. This new general presentation not only unifies previously scattered insights but also makes the results applicable across a broad range of domains in which changes in the expectation due to variations of the underlying probability measures are relevant. I-B Applications The study of the variation of the integral (expectation) of h (for some fixed x\in\mathbb{R}^{n} ) due to a measure change from P_{2} to P_{1} , i.e., the value \mathsf{G}_{h}\left(x,P_{1},P_{2}\right) in ( 1 ), plays a central role in the definition of integral probability metrics (IPMs) [ 9 , 10 ] . Using the notation in ( 1 ), an IPM results from the optimization problem \displaystyle\sup_{h\in\mathcal{H}}\left|\mathsf{G}_{h}\left(x,P_{1},P_{2}\right)\right|, (9) for some fixed x\in\mathbb{R}^{n} and a particular class of functions \mathcal{H} . Note for instance that the maximum mean discrepancy is an IPM [ 11 ] , as well as the Wasserstein distance of order one [ 12 , 13 , 14 , 15 ] . Other areas of mathematics in which the variation \mathsf{G}_{h}\left(x,P_{1},P_{2}\right) in ( 1 ) plays a key role is distributionally robust optimization (DRO) [ 16 , 17 ] and optimization with relative entropy regularization [ 5 , 6 ] . In these areas, the variation \mathsf{G}_{h}\left(x,P_{1},P_{2}\right) is a central tool. See for instance, [ 18 , 4 ] . Variations of the form \mathsf{G}_{h}\left(x,P_{1},P_{2}\right) in ( 1 ) have also been studied in [ 7 ] and [ 8 ] in the particular case of statistical machine learning for the analysis of generalization error. The central observation is that the generalization error of machine learning algorithms can be written in the form \bar{\mathsf{G}}_{h}\left(P_{Y},P_{Y|X},P_{X}\right) in ( 8 ). This observation is the main building block of the method of gaps introduced in [ 8 ] , which leads to a number of closed-form expressions for the generalization error involving mutual information, lautum information, among other information measures. II Preliminaries The main results presented in this work involve Gibbs conditional probability measures. Such measures are parametrized by a Borel measurable function h:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R} ; a \sigma -finite measure Q on \mathbb{R}^{m} ; and a vector x\in\mathbb{R}^{n} . Note that the variable x will remain inactive until Section IV . Although it is introduced now for consistency, it could be removed altogether from all results presented in this section and Section III . Denote by \mathsf{K}_{h,Q,x}:\mathbb{R}\to\mathbb{R} the function that satisfies \displaystyle\mathsf{K}_{h,Q,x}\left(t\right) \displaystyle= \displaystyle\log\left(\int\exp\left(t\,h(x,y)\right)\mathrm{d}Q\left(y\right)\right). (10) Under the assumption that Q is a probability measure, the function \mathsf{K}_{h,Q,x} in ( 10 ) is the cumulant generating function of the random variable h(x,Y) , for some fixed x\in\mathbb{R}^{n} and Y\sim Q . Using this notation, the definition of the Gibbs conditional probability measure is presented hereunder. Definition 2 (Gibbs Conditional Probability Measure) . Given a Borel measurable function h:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R} ; a \sigma -finite measure Q on \mathbb{R}^{m} ; and a \lambda\in\mathbb{R} , the probability measure P^{(h,Q,\lambda)}_{Y|X}\in\triangle\left(\mathbb{R}^{m}|\mathbb{R}^{n}\right) is said to be an (h,Q,\lambda) -Gibbs conditional probability measure if \displaystyle\mbox{$\forall x\in\mathbb{R}^{n}$, }\mathsf{K}_{h,Q,x}\left(-\lambda\right) \displaystyle< \displaystyle+\infty; (11) and for all (x,y)\in\mathbb{R}^{n}\times\mathbb{R}^{m} , \displaystyle\frac{\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}}{\mathrm{d}Q}\left(y\right) \displaystyle= \displaystyle\exp\left(-\lambda h\left(x,y\right)-\mathsf{K}_{h,Q,x}\left(-\lambda\right)\right), (12) where the function \mathsf{K}_{h,Q,x} is defined in ( 10 ). Note that, while P_{Y|X}^{(h,Q,\lambda)} is an (h,Q,\lambda) -Gibbs conditional probability measure, the measure P_{Y|X=x}^{(h,Q,\lambda)} , obtained by conditioning it upon a given vector x\in\mathbb{R}^{n} , is referred to as an (h,Q,\lambda) -Gibbs probability measure. The condition in ( 11 ) is easily met under certain assumptions. For instance, if h is a nonnegative function and Q is a finite measure, then it holds for all \lambda\in\left(-\infty,0\right) . Let \triangle_{Q}\left(\mathbb{R}^{m}\right)\triangleq\left\{P\in\triangle\left(\mathbb{R}^{m}\right):P\ll Q\right\} , with P\ll Q standing for “ P absolutely continuous with respect to Q ”. The relevance of (h,Q,\lambda) -Gibbs probability measures relies on the fact that under some conditions, they are the unique solutions to problems of the form, \displaystyle\min_{P\in\triangle_{Q}(\mathbb{R}^{m})}\int h(x,y)\mathrm{d}P(y)+\frac{1}{\lambda}D\left(P\middle\|Q\right),\mbox{ and } (13) \displaystyle\max_{P\in\triangle_{Q}(\mathbb{R}^{m})}\int h(x,y)\mathrm{d}P(y)+\frac{1}{\lambda}D\left(P\middle\|Q\right), (14) where \lambda\in\mathbb{R}\setminus\{0\} , x\in\mathbb{R} , and D\left(P\middle\|Q\right) denotes the relative entropy (or KL divergence) of P with respect to Q . The connection between the optimization problem ( 13 ) and the Gibbs probability measure P^{(h,Q,\lambda)}_{Y|X=x} in ( 12 ) appeared for the first time in [ 5 , Theorem 3 ] . Alternatively, the connection between the optimization problem ( 14 ) and such a Gibbs probability measure appeared first in [ 7 , Theorem 1 ] . In both cases, the presentation was in the context of a statistical learning problem. A general and unified statement of these observations is presented hereunder. Lemma 1 . Assume that the optimization problem in ( 13 ) (respectively, in ( 14 )) admits a solution. Then, if \lambda>0 (respectively, if \lambda<0 ), the probability measure P^{(h,Q,\lambda)}_{Y|X=x} in ( 12 ) is the unique solution. Proof: For the case in which \lambda>0 , the proof follows the same approach as the proof of [ 5 , Theorem 3 ] . Alternatively, for the case in which \lambda<0 , the proof follows along the lines of the proof of [ 7 , Theorem 1 ] . ∎ The following lemma highlights a key property of (h,Q,\lambda) -Gibbs conditional probability measures. A special case of this result was first introduced in [ 7 , Lemma 3] for the case in which \lambda<0 . Nonetheless, it holds in general for all \lambda\in\mathbb{R}\setminus\{0\} , as shown by the following lemma. Lemma 2 . Given an (h,Q,\lambda) -Gibbs probability measure, denoted by P^{(h,Q,\lambda)}_{Y|X=x} , with x\in\mathbb{R}^{n} , \displaystyle-\frac{1}{\lambda}\mathsf{K}_{h,Q,x}\left(-\lambda\right)=\int h(x,y)\mathrm{d}Q\left(y\right)-\frac{1}{\lambda}D\left(Q\middle\|P^{(h,Q,\lambda)}_{Y|X=x}\right) (15) \displaystyle= \displaystyle\int h(x,y)\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}\left(y\right)+\frac{1}{\lambda}D\left(P^{(h,Q,\lambda)}_{Y|X=x}\middle\|Q\right); (16) moreover, if \lambda>0 , \displaystyle-\frac{1}{\lambda}\mathsf{K}_{h,Q,x}\left(-\lambda\right) \displaystyle= \displaystyle\min_{P\in\triangle_{Q}(\mathbb{R}^{m})}\int h(x,y)\mathrm{d}P(y)+\frac{1}{\lambda}D\left(P\middle\|Q\right); (17) alternatively, if \lambda<0 , \displaystyle-\frac{1}{\lambda}\mathsf{K}_{h,Q,x}\left(-\lambda\right) \displaystyle= \displaystyle\max_{P\in\triangle_{Q}(\mathbb{R}^{m})}\int h(x,y)\mathrm{d}P(y)+\frac{1}{\lambda}D\left(P\middle\|Q\right), (18) where the function \mathsf{K}_{h,Q,x} is defined in ( 10 ). Proof: The proof of ( 16 ) follows from taking the logarithm of both sides of ( 12 ) and integrating with respect to P^{(h,Q,\lambda)}_{Y|X=x} . As for the proof of ( 15 ), it follows by noticing that for all (x,y)\in\mathbb{R}^{n}\times\supp Q , the Radon-Nikodym derivative \frac{\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}}{\mathrm{d}Q}\left(y\right) in ( 12 ) is strictly positive. Thus, \frac{\mathrm{d}Q}{\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}}\left(y\right)=\left(\frac{\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}}{\mathrm{d}Q}\left(y\right)\right)^{-1} . Hence, taking the negative logarithm of both sides of ( 12 ) and integrating with respect to Q leads to ( 15 ). Finally, the equalities in ( 17 ) and ( 18 ) follow from Lemma 1 and ( 16 ). ∎The following lemma introduces the main building block of this work, which is a characterization of the variation \mathsf{G}_{h}\left(x,P,P^{(h,Q,\lambda)}_{Y|X=x}\right) . Such a result appeared for the first time in [ 4 , Theorem 1] for the case in which \lambda>0 ; and in [ 7 , Theorem 6] for the case in which \lambda<0 , in different contexts of statistical machine learning. A general and unified statement of such results is presented hereunder. Lemma 3 . Consider an (h,Q,\lambda) -Gibbs probability measure, denoted by P^{(h,Q,\lambda)}_{Y|X=x}\in\triangle\left(\mathbb{R}^{m}\right) , with \lambda\neq 0 and x\in\mathbb{R} . For all P\in\triangle_{Q}\left(\mathbb{R}^{m}\right) , \displaystyle\mathsf{G}_{h}\left(x,P,P^{(h,Q,\lambda)}_{Y|X=x}\right) (19) \displaystyle= \displaystyle\frac{1}{\lambda}\left(D\left(P\middle\|P^{(h,Q,\lambda)}_{Y|X=x}\right)+D\left(P^{(h,Q,\lambda)}_{Y|X=x}\middle\|Q\right)-D\left(P\middle\|Q\right)\right). Proof: The proof follows along the lines of the proofs of [ 4 , Theorem 1] for the case in which \lambda>0 ; and in [ 7 , Theorem 6] for the case in which \lambda<0 . A unified proof is presented hereunder by noticing that for all P\in\triangle_{Q}\left(\mathbb{R}^{m}\right) , \displaystyle D\left(P\middle\|P^{(h,Q,\lambda)}_{Y|X=x}\right) (20) \displaystyle= \displaystyle\int\log\left(\frac{\mathrm{d}P}{\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}}(y)\right)\mathrm{d}P(y) \displaystyle= \displaystyle\int\log\left(\frac{\mathrm{d}Q}{\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}}(y)\frac{\mathrm{d}P}{\mathrm{d}Q}(y)\right)\mathrm{d}P(y) (21) \displaystyle= \displaystyle\int\log\left(\frac{\mathrm{d}Q}{\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}}(y)\right)\mathrm{d}P(y)+D\left(P\middle\|Q\right) (22) \displaystyle= \displaystyle\lambda\int h(x,y)\mathrm{d}P(y)+\mathsf{K}_{h,Q,x}\left(-\lambda\right)+D\left(P\middle\|Q\right) (23) \displaystyle= \displaystyle\lambda\mathsf{G}_{h}\left(x,P,P^{(h,Q,\lambda)}_{Y|X=x}\right)-D\left(P^{(h,Q,\lambda)}_{Y|X=x}\middle\|Q\right)+D\left(P\middle\|Q\right), (24) where ( 23 ) follows from ( 12 ); and ( 24 ) follows from ( 16 ). ∎It is interesting to highlight that \mathsf{G}_{h}\left(x,P,P^{(h,Q,\lambda)}_{Y|X=x}\right) in ( 19 ) characterizes the variation of the expectation of the function h(x,\cdot):\mathbb{R}^{m}\to\mathbb{R} , when \lambda>0 (resp. \lambda<0 ) and the probability measure changes from the solution to the optimization problem ( 13 ) (resp. ( 14 )) to an alternative measure P . Finally, when the reference measure Q is a probability measure, the converse of the Pythagorean theorem [ 19 , Book I, Proposition 48 ] together with Lemma 3 , lead to the geometric construction shown in Figure 1 . Such a geometric interpretation is similar to those presented in [ 8 , Figure 6] and [ 8 , Figure 7] in the context of the generalization error of machine learning algorithms. The former considers \lambda>0 , while the latter considers \lambda<0 . Nonetheless, the interpretation in Figure 1 is general and independent of such an application. \sqrt{D\left(P^{(h,Q,\lambda)}_{Y|X=x}\middle\|Q\right)} \sqrt{D\left(P\middle\|P^{(h,Q,\lambda)}_{Y|X=x}\right)} \sqrt{\lambda\mathsf{G}_{h}\left(x,P,P^{(h,Q,\lambda)}_{Y|X=x}\right)+D\left(P\middle\|Q\right)} Fig. 1: Geometric interpretation of Lemma 3 , with Q a probability measure. III Characterization of \mathsf{G}_{h}\left(x,P_{1},P_{2}\right) in ( 1 ) The main result of this section is the following theorem. Theorem 4 . For all probability measures P_{1} and P_{2} , both absolutely continuous with respect to a given \sigma -finite measure Q on \mathbb{R}^{m} , the variation \mathsf{G}_{h}\left(x,P_{1},P_{2}\right) in ( 1 ) satisfies, \displaystyle\mathsf{G}_{h}\left(x,P_{1},P_{2}\right) \displaystyle= \displaystyle\frac{1}{\lambda}\bigg(D\left(P_{1}\middle\|P^{\left(h,Q,\lambda\right)}_{Y|X=x}\right)-D\left(P_{2}\middle\|P^{\left(h,Q,\lambda\right)}_{Y|X=x}\right) (25) \displaystyle+D\left(P_{2}\middle\|Q\right)-D\left(P_{1}\middle\|Q\right)\bigg), where the probability measure P^{\left(h,Q,\lambda\right)}_{Y|X=x} , with \lambda\neq 0 , is an (h,Q,\lambda) -Gibbs probability measure. Proof: The proof follows from Lemma 3 and by observing that \mathsf{G}_{h}\left(x,P_{1},P_{2}\right)=\mathsf{G}_{h}\left(x,P_{1},P^{(h,Q,\lambda)}_{Y|X=x}\right)-\mathsf{G}_{h}\left(x,P_{2},P^{(h,Q,\lambda)}_{Y|X=x}\right) . ∎ Theorem 4 might be particularly simplified in the case in which the reference measure Q is a probability measure. Consider for instance the case in which P_{1}\ll P_{2} (or P_{2}\ll P_{1} ). In such a case, the reference measure might be chosen as P_{2} (or P_{1} ), as shown hereunder. Corollary 5 . Consider the variation \mathsf{G}_{h}\left(x,P_{1},P_{2}\right) in ( 1 ). If the probability measure P_{1} is absolutely continuous with respect to P_{2} , then, \displaystyle\mathsf{G}_{h}\left(x,P_{1},P_{2}\right) \displaystyle= \displaystyle\frac{1}{\lambda}\bigg(D\left(P_{1}\middle\|P^{\left(h,P_{2},\lambda\right)}_{Y|X=x}\right)-D\left(P_{2}\middle\|P^{\left(h,P_{2},\lambda\right)}_{Y|X=x}\right) (26) \displaystyle-D\left(P_{1}\middle\|P_{2}\right)\bigg). Alternatively, if the probability measure P_{2} is absolutely continuous with respect to P_{1} , then, \displaystyle\mathsf{G}_{h}\left(x,P_{1},P_{2}\right) \displaystyle= \displaystyle\frac{1}{\lambda}\bigg(D\left(P_{1}\middle\|P^{\left(h,P_{1},\lambda\right)}_{Y|X=x}\right)-D\left(P_{2}\middle\|P^{\left(h,P_{1},\lambda\right)}_{Y|X=x}\right) (27) \displaystyle+D\left(P_{2}\middle\|P_{1}\right)\bigg), where the probability measures P^{\left(h,P_{1},\lambda\right)}_{Y|X=x} and P^{\left(h,P_{2},\lambda\right)}_{Y|X=x} are respectively (h,P_{1},\lambda) - and (h,P_{2},\lambda) -Gibbs probability measures, with \lambda\neq 0 . In the case in which neither P_{1} is absolutely continuous with respect to P_{2} ; nor P_{2} is absolutely continuous with respect to P_{1} , the reference measure Q in Theorem 4 can always be chosen as a convex combination of P_{1} and P_{2} . That is, for all Borel sets \mathcal{A}\in\mathscr{B}\left(\mathbb{R}^{m}\right) , Q\left(\mathcal{A}\right)=\alpha P_{1}\left(\mathcal{A}\right)+(1-\lambda)P_{2}\left(\mathcal{A}\right) , with \alpha\in(0,1) . Theorem 4 can be especialized to the specific cases in which Q is the Lebesgue or the counting measure. If Q is the Lebesgue measure the probability measures P_{1} and P_{2} in ( 25 ) admit probability density functions f_{1} and f_{2} , respectively. Moreover, the terms -D\left(P_{1}\middle\|Q\right) and -D\left(P_{2}\middle\|Q\right) are Shannon’s differential entropies [ 1 ] induced by P_{1} and P_{2} , denoted by \mathsf{h}(P_{1}) and \mathsf{h}(P_{2}) , respectively. That is, for all i\in\{1,2\} , \mathsf{h}(P_{i})\triangleq-\int f_{i}(x)\log f_{i}(x)\mathrm{d}x . The probability measure P^{\left(h,Q,\lambda\right)}_{Y|X=x} , with \lambda\neq 0 , x\in\mathbb{R}^{n} , and Q the Lebesgue measure, possesses a probability density function, denoted by f^{\left(h,Q,\lambda\right)}_{Y|X=x}:\mathbb{R}^{m}\to(0,+\infty) , which satisfies \displaystyle f^{\left(h,Q,\lambda\right)}_{Y|X=x}(y) \displaystyle= \displaystyle\frac{\exp\left(-\lambda h(x,y)\right)}{\int\exp\left(-\lambda h(x,y)\right)\mathrm{d}y}. (28) If Q is the counting measure the probability measures P_{1} and P_{2} in ( 25 ) admit probability mass functions p_{1}:\mathcal{Y}\to[0,1] and p_{2}:\mathcal{Y}\to[0,1] , with \mathcal{Y} a countable subset of \mathbb{R}^{m} . Moreover, -D\left(P_{1}\middle\|Q\right) and -D\left(P_{2}\middle\|Q\right) are respectively Shannon’s discrete entropies [ 1 ] induced by P_{1} and P_{2} , denoted by \mathsf{H}(P_{1}) and \mathsf{H}(P_{2}) , respectively. That is, for all i\in\{1,2\} , \displaystyle\mathsf{H}(P_{i}) \displaystyle\triangleq \displaystyle-\sum_{y\in\mathcal{Y}}p_{i}(y)\log p_{i}(y). (29) The probability measure P^{\left(h,Q,\lambda\right)}_{Y|X=x} , with \lambda\neq 0 and Q the counting measure, possesses a conditional probability mass function, denoted by p^{\left(h,Q,\lambda\right)}_{Y|X=x}:\mathcal{Y}\to(0,+\infty) , which satisfies \displaystyle p^{\left(h,Q,\lambda\right)}_{Y|X=x}(y) \displaystyle= \displaystyle\frac{\exp\left(-\lambda h(x,y)\right)}{\sum_{y\in\mathcal{Y}}\exp\left(-\lambda h(x,y)\right)}. (30) IV Characterizations of \bar{\mathsf{G}}_{h}\left(P_{Y|X}^{(1)},P_{Y|X}^{(2)},P_{X}\right) in ( 5 ) The main result of this section is a characterization of \bar{\mathsf{G}}_{h}\left(P_{Y|X}^{(1)},P_{Y|X}^{(2)},P_{X}\right) in ( 5 ). Theorem 6 . Consider the variation \bar{\mathsf{G}}_{h}\left(P_{Y|X}^{(1)},P_{Y|X}^{(2)},P_{X}\right) in ( 5 ) and assume that for all x\in\mathbb{R}^{n} , the probability measures P_{Y|X=x}^{(1)} and P_{Y|X=x}^{(2)} are both absolutely continuous with respect to a \sigma -measure Q . Then, \displaystyle\bar{\mathsf{G}}_{h}\left(P_{Y|X}^{(1)},P_{Y|X}^{(2)},P_{X}\right) (31) \displaystyle= \displaystyle\frac{1}{\lambda}\int\bigg(D\left(P_{Y|X=x}^{(1)}\middle\|P^{\left(h,Q,\lambda\right)}_{Y|X=x}\right)-D\left(P_{Y|X=x}^{(2)}\middle\|P^{\left(h,Q,\lambda\right)}_{Y|X=x}\right) \displaystyle+D\left(P_{Y|X=x}^{(2)}\middle\|Q\right)-D\left(P_{Y|X=x}^{(1)}\middle\|Q\right)\bigg)\mathrm{d}P_{X}(x), where the probability measure P^{\left(h,Q,\lambda\right)}_{Y|X} , with \lambda\neq 0 , is an (h,Q,\lambda) -Gibbs conditional probability measure. Proof: The proof follows from ( 5 ) and Theorem 4 . ∎ Note that, from ( 5 ), it follows that the general expression for the expected variation \bar{\mathsf{G}}_{h}\left(P_{Y|X}^{(1)},P_{Y|X}^{(2)},P_{X}\right) might be simplified according to Corollary 5 . For instance, if for all x\in\mathbb{R}^{m} , the probability measure P_{Y|X=x}^{(1)} is absolutely continuous with respect to P_{Y|X=x}^{(2)} , the measure P_{Y|X=x}^{(2)} can be chosen to be the reference measure in the calculation of \mathsf{G}_{h}\left(x,P_{Y|X=x}^{(1)},P_{Y|X=x}^{(2)}\right) in ( 5 ). This observation leads to the following corollary of Theorem 6 . Corollary 7 . Consider the variation \bar{\mathsf{G}}_{h}\left(P_{Y|X}^{(1)},P_{Y|X}^{(2)},P_{X}\right) in ( 5 ) and assume that for all x\in\mathbb{R}^{n} , P_{Y|X=x}^{(1)}\ll P_{Y|X=x}^{(2)} . Then, \displaystyle\bar{\mathsf{G}}_{h}\left(P_{Y|X}^{(1)},P_{Y|X}^{(2)},P_{X}\right) (32) \displaystyle= \displaystyle\frac{1}{\lambda}\int\Bigg(D\left(P_{Y|X=x}^{(1)}\middle\|P^{\left(h,P_{Y|X=x}^{(2)},\lambda\right)}_{Y|X=x}\right)-D\left(P_{Y|X=x}^{(2)}\middle\|P^{\left(h,P_{Y|X=x}^{(2)},\lambda\right)}_{Y|X=x}\right) \displaystyle-D\left(P_{Y|X=x}^{(1)}\middle\|P_{Y|X=x}^{(2)}\right)\Bigg)\mathrm{d}P_{X}(x). Alternatively, if for all x\in\mathbb{R}^{n} , the probability measure P_{Y|X=x}^{(2)} is absolutely continuous with respect to P_{Y|X=x}^{(1)} , then, \displaystyle\bar{\mathsf{G}}_{h}\left(P_{Y|X}^{(1)},P_{Y|X}^{(2)},P_{X}\right)=\frac{1}{\lambda}\int\Bigg(D\left(P_{Y|X=x}^{(1)}\middle\|P^{\left(h,P_{Y|X=x}^{(1)},\lambda\right)}_{Y|X=x}\right) \displaystyle-D\left(P_{Y|X=x}^{(2)}\middle\|P^{\left(h,P_{Y|X=x}^{(1)},\lambda\right)}_{Y|X=x}\right)+D\left(P_{Y|X=x}^{(2)}\middle\|P_{Y|X=x}^{(1)}\right)\Bigg)\mathrm{d}P_{X}(x), (33) where the measures P^{\left(h,P_{Y|X=x}^{(1)},\lambda\right)}_{Y|X=x} and P^{\left(h,P_{Y|X=x}^{(2)},\lambda\right)}_{Y|X=x} are (h,P_{Y|X=x}^{(1)},\lambda) - and (h,P_{Y|X=x}^{(2)},\lambda) -Gibbs probability measures, respectively. The Gibbs probability measures P^{\left(h,P_{Y|X=x}^{(1)},\lambda\right)}_{Y|X=x} and P^{\left(h,P_{Y|X=x}^{(2)},\lambda\right)}_{Y|X=x} in Corollary 7 are particularly interesting as their reference measures depend on x . Gibbs measures of this form appear, for instance, in [ 5 , Corollary 10 ] . Two special cases are particularly noteworthy. When the reference measure Q is the Lebesgue measure observe that the terms -\int D\left(P_{Y|X=x}^{(1)}\middle\|Q\right)\mathrm{d}P_{X}(x) and -\int D\left(P_{Y|X=x}^{(2)}\middle\|Q\right)\mathrm{d}P_{X}(x) in ( 31 ) both become Shannon’s differential conditional entropy, denoted by \mathsf{h}\left(P_{Y|X}^{(1)}|P_{X}\right) and \mathsf{h}\left(P_{Y|X}^{(2)}|P_{X}\right) , respectively. That is, for all i\in\{1,2\} , \mathsf{h}\left(P_{Y|X}^{(i)}|P_{X}\right)\triangleq\int\mathsf{h}\left(P_{Y|X=x}^{(i)}\right)\mathrm{d}P_{X}(x), where \mathsf{h} is the entropy functional in ( ). When the reference measure Q is the counting measure the terms -\int D\left(P_{Y|X=x}^{(1)}\middle\|Q\right)\mathrm{d}P_{X}(x) and -\int D\left(P_{Y|X=x}^{(2)}\middle\|Q\right)\mathrm{d}P_{X}(x) in ( 31 ) both become Shannon’s discrete conditional entropies, denoted by \mathsf{H}\left(P_{Y|X}^{(1)}|P_{X}\right) and \mathsf{H}\left(P_{Y|X}^{(2)}|P_{X}\right) , respectively. That is, for all i\in\{1,2\} , \displaystyle\mathsf{H}\left(P_{Y|X}^{(i)}|P_{X}\right) \displaystyle\triangleq \displaystyle\int\mathsf{H}\left(P_{Y|X=x}^{(i)}\right)\mathrm{d}P_{X}(x), (34) where \mathsf{H} is the entropy functional in ( 29 ). V Characterizations of \bar{\mathsf{G}}_{h}\left(P_{Y},P_{Y|X},P_{X}\right) in ( 8 ) The main result of this section is a characterization of \bar{\mathsf{G}}_{h}\left(P_{Y},P_{Y|X},P_{X}\right) in ( 8 ), which describes the variation of the expectation of the function h when the probability measure changes from the joint probability measure P_{Y|X}P_{X} to the product of its marginals P_{Y}\cdot P_{X} . This result is presented hereunder and involves the mutual information I\left(P_{Y|X};P_{X}\right) and lautum information L\left(P_{Y|X};P_{X}\right) , defined as follows: \displaystyle I\left(P_{Y|X};P_{X}\right) \displaystyle\triangleq \displaystyle\int D\left(P_{Y|X=x}\middle\|P_{Y}\right)\mathrm{d}P_{X}(x);\mbox{ and } (35) \displaystyle L\left(P_{Y|X};P_{X}\right) \displaystyle\triangleq \displaystyle\int D\left(P_{Y}\middle\|P_{Y|X=x}\right)\mathrm{d}P_{X}(x). (36) Theorem 8 . Consider the expected variation \bar{\mathsf{G}}_{h}\left(P_{Y},P_{Y|X},P_{X}\right) in ( 8 ) and assume that, for all x\in\mathbb{R}^{n} : 1. The probability measures P_{Y} and P_{Y|X=x} are both absolutely continuous with respect to a given \sigma -finite measure Q ; and 2. The probability measures P_{Y} and P_{Y|X=x} are mutually absolutely continuous. Then, it follows that \displaystyle\bar{\mathsf{G}}_{h}\left(P_{Y},P_{Y|X},P_{X}\right)=\frac{1}{\lambda}\Bigg(I\left(P_{Y|X};P_{X}\right)+L\left(P_{Y|X};P_{X}\right) \displaystyle+\int\int\log\left(\frac{\mathrm{d}P_{Y|X=x}}{\mathrm{d}P^{\left(h,Q,\lambda\right)}_{Y|X=x}}(y)\right)\mathrm{d}P_{Y}(y)\mathrm{d}P_{X}(x) \displaystyle-\int\int\log\left(\frac{\mathrm{d}P_{Y|X=x}}{\mathrm{d}P^{\left(h,Q,\lambda\right)}_{Y|X=x}}(y)\right)\mathrm{d}P_{Y|X=x}(y)\mathrm{d}P_{X}(x)\Bigg), (37) where the probability measure P^{\left(h,Q,\lambda\right)}_{Y|X} , with \lambda\neq 0 , is an (h,Q,\lambda) -Gibbs conditional probability measure. Proof: The proof follows from Theorem 6 , which holds under assumption (a) and leads to \displaystyle\bar{\mathsf{G}}_{h}\left(P_{Y},P_{Y|X},P_{X}\right) (38) \displaystyle= \displaystyle\frac{1}{\lambda}\int\bigg(D\left(P_{Y}\middle\|P^{\left(h,Q,\lambda\right)}_{Y|X=x}\right)-D\left(P_{Y|X=x}\middle\|P^{\left(h,Q,\lambda\right)}_{Y|X=x}\right) \displaystyle+D\left(P_{Y|X=x}\middle\|Q\right)-D\left(P_{Y}\middle\|Q\right)\bigg)\mathrm{d}P_{X}(x). The proof continues by noticing that \displaystyle\int D\left(P_{Y|X=x}\middle\|Q\right)\mathrm{d}P_{X}(x) \displaystyle= \displaystyle I\left(P_{Y|X};P_{X}\right)+D\left(P_{Y}\middle\|Q\right), (39) and \displaystyle\int D\left(P_{Y}\middle\|P^{\left(h,Q,\lambda\right)}_{Y|X=x}\right)\mathrm{d}P_{X}(x)=L\left(P_{Y|X};P_{X}\right) \displaystyle+\int\int\log\left(\frac{\mathrm{d}P_{Y|X=x}}{\mathrm{d}P^{\left(h,Q,\lambda\right)}_{Y|X=x}}(y)\right)\mathrm{d}P_{Y}(y)\mathrm{d}P_{X}(x). (40) Finally, using ( 39 ) and ( 40 ) in ( 38 ) yields ( 37 ), which completes the proof. ∎ An interesting observation from Theorem 8 is that the last two terms in the right-hand side of ( 37 ) are both zero in the case in which P_{Y|X} is an (h,Q,\lambda) -Gibbs conditional probability measure. This is observation is highlighted by the following corollary. Corollary 9 . Consider an (h,Q,\lambda) -Gibbs conditional probability measure, denoted by P^{(h,Q,\lambda)}_{Y|X}\in\triangle\left(\mathbb{R}^{m}|\mathbb{R}^{n}\right) , with \lambda\neq 0 ; and a probability measure P_{X}\in\triangle\left(\mathbb{R}^{n}\right) . Let the measure P^{(h,Q,\lambda)}_{Y}\in\triangle\left(\mathbb{R}^{m}\right) be such that for all sets \mathcal{A}\in\mathscr{B}\left(\mathbb{R}^{m}\right) , \displaystyle P^{(h,Q,\lambda)}_{Y}\left(\mathcal{A}\right) \displaystyle= \displaystyle\int P^{(h,Q,\lambda)}_{Y|X=x}\left(\mathcal{A}\right)\mathrm{d}P_{X}(x). (41) Then, \displaystyle\bar{\mathsf{G}}_{h}\left(P^{(h,Q,\lambda)}_{Y},P^{(h,Q,\lambda)}_{Y|X},P_{X}\right) (42) \displaystyle= \displaystyle\frac{1}{\lambda}\left(I\left(P^{(h,Q,\lambda)}_{Y|X};P_{X}\right)+L\left(P^{(h,Q,\lambda)}_{Y|X};P_{X}\right)\right).