Numerical evaluation of Gaussian mixture entropy

Basheer Joudeh, Boris Škorić · 2025

Read Numerical evaluation of Gaussian mixture entropy by Basheer Joudeh, Boris Škorić (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14047.txt.

Introduction

Basheer Joudeh and Boris Škorić build a numerical method for the entropy of a Gaussian mixture, in work from 2025. The differential entropy of such a mixture has no closed form, yet it is needed in signal processing and learning. Their scheme produces two approximations from one frame, named for the tools each uses, a Taylor route and a polynomial fit route.

The Taylor version is easy to compute and always sits below the true entropy, giving a safe lower bound. The fit version is both fast and accurate, beating known bounds in their tests. The authors also conjecture it holds up far better than a prior method as dimension grows.

Catalog record

CREATORBasheer Joudeh, Boris Škorić
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14047

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Numerical evaluation of Gaussian mixture entropy Basheer Joudeh, Boris Škorić Abstract We develop an approximation method for the differential entropy h ⁡ ( 𝐗 ) h({\mathbf{X}}) of a q q -component Gaussian mixture in ℝ n \mathbb{R}^{n} . We provide two examples of approximations using our method denoted by h ¯ C , m Taylor ​ ( 𝐗 ) \bar{h}^{\mathrm{Taylor}}_{C,m}({\mathbf{X}}) and h ¯ C Polyfit ​ ( 𝐗 ) \bar{h}^{\mathrm{Polyfit}}_{C}({\mathbf{X}}) . We show that h ¯ C , m Taylor ​ ( 𝐗 ) \bar{h}^{\mathrm{Taylor}}_{C,m}({\mathbf{X}}) provides an easy to compute lower bound to h ⁡ ( 𝐗 ) h({\mathbf{X}}) , while h ¯ C Polyfit ​ ( 𝐗 ) \bar{h}^{\mathrm{Polyfit}}_{C}({\mathbf{X}}) provides an accurate and efficient approximation to h ⁡ ( 𝐗 ) h({\mathbf{X}}) . h ¯ C Polyfit ​ ( 𝐗 ) \bar{h}^{\mathrm{Polyfit}}_{C}({\mathbf{X}}) is more accurate than known bounds, and conjectured to be much more resilient than the approximation of [ 5 ] in high dimensions. 1 Introduction 1.1 Differential entropy of Gaussian mixtures A Gaussian mixture is a probability density function on {\mathbb{R}}^{n} of the form f({\mathbf{x}})=\sum_{j=1}^{q}p_{j}{\cal N}_{{\mathbf{w}}_{j},K_{j}}({\mathbf{x}}) , where {\mathbf{x}}\in{\mathbb{R}}^{n} , the p_{j}>0 are weights satisfying \sum_{j=1}^{q}p_{j}=1 , and {\cal N}_{{\mathbf{w}},K} stands for a Gaussian distribution with mean {\mathbf{w}} and covariance matrix K . In words, f is a mixture of q Gaussian pdfs, with arbitrary weights, allowing each Gaussian to be of different shape and displacement. A mixture like this occurs e.g. when a stochastic process, with probability mass function p_{1},\ldots,p_{q} , determines which distribution holds for {\mathbf{x}} . Such situations occur in many scientific areas. Furthermore, Gaussian mixtures are often employed as function approximators, by virtue of being smooth and localized. They have been used in a wide variety of studies, e.g. on diffusion models in physics and machine learning [ 4 , 17 , 2 , 13 ] , non-adiabatic thermodynamics [ 12 ] , wireless communication [ 15 ] , wireless authentication [ 11 ] , Byzantine attacks [ 10 ] , gene expression [ 7 , 14 ] , dark matter kinematics [ 18 ] , and quasar spectra [ 16 ] . The differential entropy h(X) of a continuous random variable X\in{\cal X} , with probability density function f_{X} , is defined as h(X)=-\int_{\cal X}{\rm d}x\;f_{X}(x)\ln f_{X}(x) . It represents the continuum limit of the (discrete) Shannon entropy for the probability mass function f_{X}(x_{i})\triangle x , where the volume \triangle x is sent to zero and the infinite contribution \ln\frac{1}{\triangle x} is subtracted. Analytically computing or estimating the differential entropy of a Gaussian mixture is a notoriously difficult problem. Even numerical evaluation can be problematic in high dimensions. The special case of a single Gaussian is simple and yields h(X)=\frac{1}{2}\ln\det(2\pi eK) . 1.2 Related work Various methods have been proposed to approximate the differential entropy of a Gaussian mixture. A loose upper bound can be obtained from the fact that a Gaussian distribution maximizes entropy, given the first and second moment. This yields [ 9 ] h(X)\leq\frac{1}{2}\ln\det(2\pi e\Sigma) , with \Sigma=\sum_{i=1}^{q}p_{i}(w_{i}w_{i}^{\rm T}+K_{i})-\left(\sum_{i=1}^{q}p_{i}w_{i}\right)(\sum_{j=1}^{q}p_{j}w_{j})^{\rm T} . In [ 8 ] a numerical approximation was given for the case q=2,n=1,w_{2}=-w_{1} . A sequence of upper bounds for arbitrary q was obtained in [ 9 ] , but only for n=1 . Furthermore, the results are not in closed form, and the bounds in the sequence do not get progressively tighter. Another method is to replace the density f inside the logarithm by a single Gaussian \bar{f} which has mean and covariance equal to the mixture. This leads to an exact expression containing the relative entropy, h(X)=\frac{1}{2}\ln\det(2\pi e\Sigma)-D(f||\bar{f}) ; one can then find approximations or bounds for the relative entropy, as in [ 3 , 1 ] . This is done with Monte Carlo integration, which has the drawback of being computationally demanding and not giving an analytic expression. In [ 5 ] an approximation was obtained by performing a Taylor expansion of \ln f({\mathbf{x}}) in the variables {\mathbf{x}}-{\mathbf{w}}_{j} . To avoid the need for expansion powers above 2, they introduced the trick of representing wide Gaussians approximately as the sum of several narrow Gaussians, in the f outside the logarithm. The Taylor expansion as well as the splitting trick introduce inaccuracies. As shown in [ 5 ] , one can obtain a basic upper bound h(X)\leq\sum_{j=1}^{q}p_{j}\ln\frac{1}{p_{j}}+\sum_{j=1}^{q}p_{j}\frac{1}{2}\ln\det(2\pi eK_{j}) . This is significantly tighter than the above mentioned bound \frac{1}{2}\ln[(2\pi e)^{n}\det\Sigma] ; it is exact in the case of a single Gaussian, and it gets arbitrarily close to the true value of h(X) when the overlap between the components of the mixture becomes negligible. A refinement was also introduced by means of merging parts of the mixture that are clustered together. However, in configurations where the mixture components have significant overlap whilst keeping the mixture non-trivial, this bound becomes inaccurate. In [ 6 ] a method was proposed to compute h(X) averaged over the means {\mathbf{w}}_{1},\ldots,{\mathbf{w}}_{q} . This is of limited use when the means are fixed. 1.3 Contributions We introduce a new method for estimating the differential entropy of a Gaussian mixture. We approximate f\ln\frac{1}{f} by a polynomial {\rm poly}(f) . For any positive integer k , the power f^{k}=(p_{1}{\cal N}_{1}+\ldots+p_{q}{\cal N}_{q})^{k} can be written as a multinomial sum containing a product of powers of Gaussians. Hence the integral \int\!{\rm d}x\;{\rm poly}(f(x)) can be computed analytically. This yields a systematic way to approximate and/or lower bound h(X) by analytic expressions. • We consider polynomials that minimize the square error \int\!{\rm d}s\;{\rm weight}(s)[s\ln\frac{1}{s}-{\rm poly}(s)]^{2} on the range of f , with tunable function {\rm weight}(s)=s^{r} . We observe that negative r gives better results than positive r , which is to be expected since most of the volume in {\mathbb{R}}^{n} has f(x) close to zero. In particular, r\approx-2 performs best in the mixture configurations that we have studied, yielding relative errors in h(X) of less than 1% already for the degree-3 polynomial fit. • We consider the truncated Taylor series -\ln\frac{f}{m}\approx\sum_{k=1}^{\cdots}\frac{1}{k}(1-\frac{f}{m})^{k} for some constant m . This too gives rise to an approximation for f\ln\frac{1}{f} that is polynomial in f . While not as accurate as the above mentioned fit, it guarantees that each k -term has the same sign if m\geq\max_{x\in{\mathbb{R}}^{n}}f(x) , and hence gives rise to a sequence of increasingly accurate analytic lower bounds on h(X) . We observe that the relative error in h(X) is still several percent even at polynomial degree 10. In Section 2.1 we obtain a general recipe to approximate the differential entropy via a polynomial approximation as shown in Corollary 1 . In Section 2.2 we apply Corollary 1 to the Taylor series of the logarithm to obtain a specific approximation for the differential entropy. Although it is not a very accurate approximation, it serves as an easy to compute lower bound. In Section 2.3 we apply Corollary 1 again using a polynomial approximation of f(s)=-s\ln s , which gives an accurate approximation to the differential entropy. In Section 3 we show numerical results of our approximations for various configurations, and in Section 4 we assess our method and give pointers towards future work. 2 Polynomial approximation 2.1 General polynomial Consider a random variable \mathbf{X}\in{\mathbb{R}}^{n} whose probability density function is a Gaussian mixture with weights \{p_{i}\}_{i=1}^{q} . The Gaussian pdfs

Read the complete text