The Entropy of Floating-Point Numbers

Sultan Daniels * , Samuel H. D’Ambrosia * , Michael R. DeWeese, and Anant Sahai · 2026

Read The Entropy of Floating-Point Numbers by Sultan Daniels * , Samuel H. D’Ambrosia * , Michael R. DeWeese, and Anant Sahai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11546.txt.

Introduction

Daniels, D'Ambrosia, DeWeese and Sahai give an analytic estimate for the entropy of floating-point numbers, with bounds on its error. Differential entropy links tightly to the discrete entropy of a uniformly rounded random variable. Their approximation uncovers a different quantity that provides that link for floating-point quantization. They also prove floating-point entropy stays nearly unchanged under scaling. Closed-form expressions for common distributions are provided and checked against exact results.

Catalog record

CREATORSultan Daniels * , Samuel H. D’Ambrosia * , Michael R. DeWeese, and Anant Sahai
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63127
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11546

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Entropy of Floating-Point Numbers Sultan Daniels * , Samuel H. D’Ambrosia * , Michael R. DeWeese, and Anant Sahai Abstract Here we present an analytic approximation for the entropy of floating-point numbers, along with bounds on the error of this approximation. It is well-known that the differential entropy is tightly linked to the discrete entropy of a uniformly quantized random variable. Our approximation uncovers a different quantity that provides this link for floating-point quantization. Additionally, we prove that the entropy of a floating-point quantized random variable is approximately unchanged under scaling. Closed-form expressions for the floating-point entropy of common distributions are provided and compared to exact results. * SMD and SHD contributed equally to this work. This work was supported in part by the H2H8 Nonprofit Organization. Approximate entropy: \tilde{H}_{s}(X_{fp})=p-\tfrac{1}{2}-\mathbb{E}_{f_{X}\!\!}\left[\log(|X|f_{X}(X))\right] X\sim\mathcal{N}(0,\sigma^{2}) \tilde{H}_{s}(X_{fp})=p+\tfrac{1}{2}\log(2\pi e)+\tfrac{\gamma_{e}}{2\ln 2}\approx p+2.46 X\sim Uniform (-a,a) \tilde{H}_{s}(X_{fp})=p+\tfrac{1}{2}+\log e\approx p+1.94 X\sim Gamma (\alpha,\theta) \tilde{H}_{s}(X_{fp})=p-\tfrac{1}{2}+\alpha\log(e)\left(1-\psi(\alpha)\right)+\log\Gamma(\alpha) X\sim\chi^{2}(k) \tilde{H}_{s}(X_{fp})=p-\tfrac{1}{2}+\tfrac{k}{2}\log(e)\left(1-\psi\!\left(\tfrac{k}{2}\right)\right)+\log\Gamma\!\left(\tfrac{k}{2}\right) X\sim Laplace (0,b) \tilde{H}_{s}(X_{fp})=p+\tfrac{1}{2}+\tfrac{1+\gamma_{e}}{\ln 2}\approx p+2.78 X\sim Logistic (0,s) \tilde{H}_{s}(X_{fp})=p-\tfrac{1}{2}+\tfrac{2+\gamma_{e}-\ln(\pi/2)}{\ln 2}\approx p+2.57 X\sim Weibull (\lambda,k) \tilde{H}_{s}(X_{fp})=p-\tfrac{1}{2}+\tfrac{1+\gamma_{e}}{\ln 2}-\log k\approx p+1.78-\log k X\sim Lognormal (\mu,\sigma^{2}) \tilde{H}_{s}(X_{fp})=p-\tfrac{1}{2}+\log\!\left(\sigma\sqrt{2\pi e}\right)\approx p+1.55+\log\sigma X\sim Pareto (x_{m},\alpha) \tilde{H}_{s}(X_{fp})=p-\tfrac{1}{2}+\log\!\left(\tfrac{e}{\alpha}\right)\approx p+0.94-\log\alpha X\sim Beta (\alpha,\beta) \tilde{H}_{s}(X_{fp})=p-\tfrac{1}{2}+\log B(\alpha,\beta) {}+\tfrac{(\alpha+\beta-1)\psi(\alpha+\beta)-\alpha\psi(\alpha)-(\beta-1)\psi(\beta)}{\ln 2} X\sim t_{\nu}(0,s) \tilde{H}_{s}(X_{fp})=p-\tfrac{1}{2}+\log B\!\left(\tfrac{\nu}{2},\tfrac{1}{2}\right) {}+\tfrac{\tfrac{\nu+1}{2}\psi\!\left(\tfrac{\nu+1}{2}\right)-\tfrac{\nu}{2}\psi\!\left(\tfrac{\nu}{2}\right)-\tfrac{1}{2}\psi\!\left(\tfrac{1}{2}\right)}{\ln 2} \mathbf{X}\sim\mathcal{N}\left(\mathbf{0}_{d},\Sigma\right) \tilde{H}_{s}(\mathbf{X}_{fp})=d\left(p+\tfrac{1}{2}\log(2\pi e)+\tfrac{\gamma_{e}}{2\ln 2}\right) {}+\tfrac{1}{2}\log\!\left(\tfrac{\prod_{i=1}^{d}\lambda_{i}(\Sigma)}{\prod_{i=1}^{d}\Sigma_{ii}}\right) TABLE I : Closed-form approximations for the floating-point entropy of common distributions. Here, p is the precision, \gamma_{e} is the Euler-Mascheroni constant, \Gamma(\cdot) is the gamma function, \psi(\cdot) is the digamma function, B(\cdot,\cdot) is the beta function, \mathbf{0}_{d} is the d -dimensional zero vector, and \lambda_{i}(\Sigma) is the i -th eigenvalue of the positive semidefinite covariance matrix \Sigma . For the multivariate Gaussian case, we assume scalar quantization in which each component of the random vector is independently quantized on distinct floating-point numbers. Note that for all the univariate distributions, the scale parameter cancels (see Theorem 3 ). Additionally, in the multivariate case, the entropy is invariant under transformation of \mathbf{X} by a diagonal matrix (see Corollary 1 ). I Introduction Floating point numbers are a common choice for the discrete representation of continuous variables [ 1 , 2 ] . This representation uses a finite register of binary bits to represent a real value in binary scientific notation. Today, floating-point numbers play a central role in modern deep learning algorithms and hardware [ 3 , 4 ] , and there has been renewed interest in developing novel floating-point formats that find the right tradeoffs between precision, dynamic range, memory, computational speed and numerical stability [ 5 , 6 ] . In the design of variable-rate quantizers, the objective is to find the best quantizer that minimizes entropy and distortion [ 7 ] . It was shown by [ 8 ] , that in the high-resolution limit, as the number of quantization levels are taken to infinity, uniform quantizers achieve this optimality. Here, we consider a fixed non-uniform quantizer: a floating-point number, and ask the basic question, what is its output entropy? We present a predictive analytic approximation for this entropy, through extending the exact relation between differential entropy and the discrete entropy of uniform quantizers with finite levels given in [ 9 ] to the setting of nonuniform quantization (Section III ), and applying a smoothing approximation for the floating-point bin size (Section IV ). This allows for closed-form expressions for the entropy of common distributions represented on floating-point numbers (Table I ), which can be compared to the numerically evaluated exact discrete entropy (Figure 1(d) ). Bounds on the error of these approximations are provided in Sections III-B and IV . Finally, we prove the scale-invariance property of the analog of the differential entropy for floating-point quantization (Section V ), which predicts that the output entropy of a floating point representation is left unchanged when the underlying continuous random variable is scaled by a constant. This contrasts the scale-dependence of differential entropy. (a) (b) (c) (d) Fig. 1 : Floating-point structure, and comparisons between analytic approximation and numerically evaluated exact discrete entropy . ( 1(a) ) The structure of a floating-point number where each box represents one bit. The true bin size function \Delta(x) is plotted on log-log scale for both midpoint (black solid curve) and floor quantization (blue dotted curve) with p=10 and E=4 along with the smooth approximation \Delta_{s}(x) (red dashed curve). ( 1(b) ) shows the entropy of X_{fp} which is a discrete representation of the random variable X\sim\mathcal{N}\left(0,s^{2}\right) that has been clipped and midpoint quantized onto a floating-point representation with precision p=3 and various numbers of exponent bits E . The x-axis shows the standard deviation s of the underlying continuous distribution. The horizontal red line shows the approximate entropy \tilde{H}_{s}(p) . The vertical dashed lines mark s=2^{e_{\min}} , while the vertical dotted lines mark s=2^{e_{\max}} for each E . Notice that exact entropy closely follows the approximate entropy until s approaches the overflow and underflow boundaries for each E . ( 1(c) ) Shows the exact discrete entropy of X_{fp} as p varies for each E when the underlying continuous random variable is X\sim\mathcal{N}\left(0,1\right) . The approximation \tilde{H}_{s}(p) is also plotted as the dashed red line. Notice that the curves for E\geq 4 are directly on top of each other, showing that \tilde{H}_{s}(p) is close to H(X_{fp}) when E is large enough to keep the probability of overflows and underflows low. The approximation stays close to the true entropy with a large exponent, even with low precision. ( 1(d) ) Comparison between exact entropies and Table I approximations for various distributions, in terms of a scale parameter s defined in the figure for each distribution, demonstrating excellent agreement inside the underflow and overflow range specified by 2^{-(2^{E-1}-1)}<s<2^{2^{E-1}} . II The Structure of a Floating-point Number A normalized floating-point number stores a real value x\in\mathbb{R} in the form \displaystyle x_{fp}(x)=s_{fp}(x)\times 2^{e_{fp}(x)}\times(1+m_{fp}(x)), (1) where s_{fp}(x) corresponds to the sign of the number x encoded by a single bit, e_{fp}(x) is the exponent of the number x encoded using E bits, and m_{fp}(x) is the mantissa or significand specifying the fractional significant digits of x encoded using p-1 bits, where p is referred to as the precision of the floating-point number [ 1 , 2 ] . Explicit definitions of s_{fp}(x),e_{fp}(x), and m_{fp}(x) are given in Appendix D . This creates a finite set of representable values along the real line. The floating-point format can be thought of as a nonuniform quantizer for a continuous random variable X . The quantization scheme is a map from the real line to K representable values \{u_{1},u_{2},\ldots,u_{K}\} , where u_{1}<u_{2}<\dots<u_{K} . We will refer to the resulting quantized discrete random variable as X_{fp} . In this paper, we will consider a simplified floating-point format, with rounding to the nearest representable value, that does not use subnormal numbers (as defined in [ 1 ] ). The structure of a floating-point number is shown in Fig. 1(a) . See [ 1 , 2 ] for further details on the floating-point representation. For E exponent bits, e_{fp}(x) is an integer between e_{min}=-(2^{E-1}-1) and e_{max}=2^{E-1} . Primarily controlled by e_{max} , the granular region \mathbb{U}\triangleq\left[-2^{e_{max}+1}+2^{e_{max}-p},\,\,2^{e_{max}+1}-2^{e_{max}-p}\right]^{d} , is the region of the real line that is near the representable values [ 7 ] . Let \Delta:\mathbb{U}\to\mathbb{R}_{+} encode the bin sizes of the quantization scheme for a single

Read the complete text