On Entropy-Constrained Gaussian Channel Capacity via the Moment Problem

Adway Girish *, Shlomo Shamai (Shitz) †, Emre Telatar * · 2025

Read On Entropy-Constrained Gaussian Channel Capacity via the Moment Problem by Adway Girish *, Shlomo Shamai (Shitz) †, Emre Telatar * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13814.txt.

Introduction

Adway Girish, Shlomo Shamai, and Emre Telatar study how much data a noisy channel can carry when the input signal must also stay simple, in a 2025 paper. The channel is the classic Gaussian one, where noise blurs each sent value. On top of a power limit, they add an entropy limit, which caps how spread out and complex the input can be.

Their main result comes from a matching puzzle: how many starting moments of a smooth random signal can a chunky, few-value signal copy while keeping its own entropy tiny? The answer is three. From that fact they pin down the channel's capacity when noise is strong and the entropy cap is small. The result links a pure math puzzle to a real limit in communications design.

Catalog record

CREATORAdway Girish *, Shlomo Shamai (Shitz) †, Emre Telatar *
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-13814

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Entropy-Constrained Gaussian Channel Capacity via the Moment Problem Adway Girish *, Shlomo Shamai (Shitz) †, Emre Telatar * Abstract We study the capacity of the power-constrained additive Gaussian channel with an entropy constraint at the input. In particular, we characterize this capacity in the low signal-to-noise ratio regime at small entropy. This follows as a corollary of the following general result on a moment matching problem: We show that for any continuous random variable with finite moments, the largest number of initial moments that can be matched by a discrete random variable of sufficiently small but positive entropy is three. 1 Introduction Consider an additive Gaussian noise channel with an input-output relationship given by Y=\sqrt{\mathsf{snr}}X+Z , where \mathsf{snr}>0 is the signal-to-noise ratio (SNR) of the channel and Z\sim\mathcal{N}(0,1) is a standard Gaussian random variable, independent of X . We denote the mutual information between X and Y by I(X,\mathsf{snr})\triangleq I(X;Y) , since we have Y=\sqrt{\mathsf{snr}}X+Z throughout this paper. The capacity of this channel with a power constraint on the input is given by C(\mathsf{snr})=\sup_{\mathbb{E}[X^{2}]\leq 1}I(X,\mathsf{snr})=\frac{1}{2}\log(1+\mathsf{snr}), where the supremum is over all distributions of X over \mathbb{R} . The supremum is achieved by taking X\sim\mathcal{N}(0,1) . Operationally, the capacity C(\mathsf{snr}) is the largest communication rate at which an arbitrarily small error probability can be ensured over an additive Gaussian channel of SNR equal to \mathsf{snr} . In modern cloud-based communication networks, it is a remote agent that determines the channel input, and the communication from the agent to the transmitter is rate-limited. Consequently, the channel input is entropy constrained. As a mathematical model of such a scenario, we propose to study the entropy-constrained capacity of the Gaussian channel, defined as C_{H}(h,\mathsf{snr})=\sup_{\begin{subarray}{c}\mathbb{E}[X^{2}]\leq 1,\\ H(X)\leq h\end{subarray}}I(X,\mathsf{snr}), (1) for h>0 . The operational interpretation of C_{H}(h,\mathsf{snr}) is that it is the largest communication rate at which an arbitrarily small error probability can be ensured over the Gaussian channel of SNR \mathsf{snr} using an input of entropy at most h . Note that the entropy constraint forces X to have a discrete distribution. Also note that without the power constraint \mathbb{E}[X^{2}]\leq 1 , C_{H}(h,\mathsf{snr}) would be equal to h for all h>0 . Computing the Gaussian channel capacity can be viewed as an approximation problem of random variables in the following sense: solving \sup_{X}I(X,\mathsf{snr}) is equivalent to solving \inf_{X}C(\mathsf{snr})-I(X,\mathsf{snr})=\inf_{X}D(X+Z\,\|\,X_{G}+Z) , where Z\sim\mathcal{N}(0,1) , X_{G} is Gaussian with the same mean and variance as X , and Z is independent of X and X_{G} . Thus, the problem is to approximate the Gaussian random variable X_{G} by an X that minimizes the “non-Gaussianity” D(X+Z\,\|\,X_{G}+Z) . Our main result characterizes C_{H}(h,\mathsf{snr}) in the low-SNR regime, i.e., \mathsf{snr}\to 0 , for sufficiently small h . It turns out that in this regime, the approximation problem described above essentially takes on the following form, which can be posed independently of the above, in terms of just the moments of the random variables involved. Consider the classical mathematical problem of approximating continuous distributions with discrete distributions by matching as many of their initial moments as possible , called the moment problem [ Akh65 , Sch17 ] . In particular, we are interested in identifying discrete distributions that match as many initial moments as possible while having a sufficiently small entropy . Moments determine the tail behaviour of distributions, but it is known that distributions that are visually very different can have nearly identical moment generating functions, and thus, moments [ McC94 ] . Continuous distributions can be thought of as having “infinite” entropy. Thus, a natural question to ask is whether we can replicate similar tail behaviours as infinite entropy continuous distributions using “low-entropy” discrete distributions. We show that discrete distributions with sufficiently small entropy (i.e., smaller than some positive constant depending on the continuous distribution) can match no more than three moments with any continuous distribution of finite moments. Using this result, we show that for any h<h_{2}(1/3)\approx 0.92 bits, as \mathsf{snr}\to 0 , C_{H}(h,\mathsf{snr}) is at most a constant factor times \mathsf{snr}^{4} away from C(\mathsf{snr}) . In addition to this low-SNR regime, we provide asymptotic expressions for C_{H}(h,\mathsf{snr}) in the regimes where h\to 0 and h\to\infty in Section 2 . These follow as immediate corollaries of results on the asymptotic tightness of F_{I} curves [ CPW18 ] . More details on the moment problem and the solution of its low-entropy version are in Section 3 . We conclude with some perspectives and open problems in Section 4 . 1.1 Prior work Several works have studied the effect of practical constraints on channel capacity. For example, Wu and Verdú [ WV10 ] considered a cardinality constraint on the (discrete) input distribution and showed, among other things, that the cardinality-constrained capacity approaches the unconstrained capacity C(\mathsf{snr}) exponentially fast as the cardinality goes to infinity. Surprisingly, a peak-amplitude constraint (instead of the power and entropy constraints) also results in the capacity-achieving distribution being discrete, as shown by Smith [ Smi71 ] . This spurred interest in characterizing the cardinality of the capacity-achieving distribution and bounds on the capacity with an amplitude constraint [ SS10 , DYPS19 , TKB17 ] . More recently, there have been extensions to various moment constraints and channels [ MW21 , ATS01 , LM09 , BZFD24 ] , motivated by practical setups such as optimal communication. While our motivation for the entropy constraint comes from cloud-based networks, it is worth noting that entropy constraints have become increasingly popular in the machine learning community, particularly in lossy source coding [ LZCK22 , ECK24 ] . The moment problem has also been of interest in machine learning [ LSZ15 , NGV21 ] , which is not unexpected, given that it is an approximation problem. Thus, identifying how many moments of continuous distributions can be matched by low-entropy distributions is a timely and interesting problem in its own right. The appearance of the moment problem in our context of computing the Gaussian channel capacity is also not surprising, as it is known that information measures associated with the Gaussian channel can be approximated by polynomials of moments [ GSV08 , AC24 ] . For an overview of the moment problem from a mathematical perspective, refer to the recent textbook by Schmüdgen [ Sch17 ] (more references are in Section 3 ). 1.2 Notation The sequence (s_{0},s_{1},\dots) is represented by the shorthand (s_{n})_{n=0}^{\infty} . Uppercase letters (e.g X , Y , …) denote random variables (random variables). We use \mathbb{E}[X] to denote the expectation of X . D(X\,\|\,Y) denotes the KL divergence between the distributions of X and Y . The mutual information between X and Y is denoted by I(X,\mathsf{snr}) , with X and Y being the input and output of an additive Gaussian channel throughout the paper. The entropy of X is denoted by H(X) , and the binary entropy function by h_{2}(x)=-x\log x-(1-x)\log(1-x) . All logarithms are taken with respect to any fixed base; when the base is 2, the unit is bits. We write f(x)=\mathsf{O}(g(x)) if there exists a finite constant M and x_{0}>0 such that f(x)\leq Mg(x) for all |x|<x_{0} . We write f(x)=\Theta(g(x)) if f(x)=\mathsf{O}(g(x)) and g(x)=\mathsf{O}(f(x)) . The Hankel matrix of order n is denoted by \mathsf{H}_{n}(s_{0},s_{1},\dots,s_{2n}) and is given by the (n+1)\times(n+1)

Read the complete text