The problem of infinite information flow

Zheng Bian, Erik M. Bollt · 2025

Read The problem of infinite information flow by Zheng Bian, Erik M. Bollt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20035.txt.

Introduction

Zheng Bian and Erik Bollt tackle a paradox where standard measures say information flow is either zero or infinite. Transfer entropy is widely used to claim causal links. When one variable is a fixed function of others, it breaks.

Under mild conditions they prove a strict dichotomy. The flow measure lands at one extreme or the other. That verdict cannot rank strong versus weak flows. A discretization strategy is proposed to restore gradations. A conjectured formula estimates the relative ambiguity instead. Numerical tests support the proxy's reliability. The result warns data scientists against naive readings. It offers a steadier way to compare information flow. It is a sharp note of care for the field.

Catalog record

CREATORZheng Bian, Erik M. Bollt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20035

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The problem of infinite information flow Zheng Bian, Erik M. Bollt Abstract We study conditional mutual information (cMI) between a pair of variables X , Y X,Y given a third one Z Z and derived quantities including transfer entropy (TE) and causation entropy (CE) in the dynamically relevant context where X = T ⁡ ( Y , Z ) X=T(Y,Z) is determined by Y , Z Y,Z via a deterministic transformation T T . Under mild continuity assumptions on their distributions, we prove a zero-infinity dichotomy for cMI for a wide class of T T , which gives a yes-or-no answer to the question of information flow as quantified by TE or CE. Such an answer fails to distinguish between the relative amounts of information flow. To resolve this problem, we propose a discretization strategy and a conjectured formula to discern the relative ambiguities of the system, which can serve as a reliable proxy for the relative amounts of information flow. We illustrate and validate this approach with numerical evidence. 1 Introduction Quantifying information flow is a critical task for understanding complex systems in various scientific disciplines, from neuroscience [ VWLP11 , URM20 , SSL21 ] to financial markets [ DP13 , ABD22 ] . Information measures such as mutual information (MI), conditional mutual information (cMI) [ CT05 ] , transfer entropy (TE) [ Sch00 ] , and causation entropy (CE) [ STB15 ] , have become essential tools for this purpose. Tracing back to the classic Weiner-Granger causality [ Gra69 , Gra88 , BBS09 , Hen04 ] , a central idea that underlies these information theoretic methods of quantifying information flow is the notion of disambiguation in a predictive framework. In contrast to the experimentalist approach, which infers causality from outcomes of perturbations and experiments, the predictive framework, which we consider below, is premised on alternative formulations of the forecasting question, with and without considering the influence of an external system. Formulated by Schreiber [ Sch00 ] in 2000, TE is a quantitative attempt in this predictive framework. We think of V=\{V_{t}\} and U=\{U_{t}\} as stochastic processes indexed by discrete time t=0,1,\cdots ; for a concrete example, imagine that V,U record EEG times series data from different parts of the brain. We expect that the present state V_{t} informs about the future state V_{t+1} and are interested in determining whether the present state U_{t} also informs about V_{t+1} . If V_{t+1} is conditionally independent of U_{t} given V_{t} , then the knowledge about the state of U_{t} does not resolve any uncertainty about the state of V_{t+1} , assuming one already has access to the state of V_{t} . In this case, we would like to conclude no information flow from U to V at time t and zero TE accordingly. Otherwise, any deviation from this conditional independence indicates the presence of information flow, to be captured and quantified by some positive value of TE measured in bits per time unit. By a slight generalization of Schreiber’s original formulation and in agreement with the usual definition for discrete variables, we define TE T_{U\to V,t}:=I(V_{t+1};U_{t}|V_{t}) (1) to be the conditional mutual information of V_{t+1},U_{t} given V_{t} . For simplicity, this is the case of lag length 1; longer lags are allowed in general. Causation entropy, proposed by Sun, Taylor and Bollt [ STB15 ] , generalizes TE to infer network connectivity [ SB14 , SCB14 , ASB20 , LSOB16 ] , by also building in conditioning on ternary influences as a way to resolve the differences between direct and indirect interactions. The precise definition of cMI will be given in Section 2 . Roughly speaking, it quantifies the deviation from conditional independence of a pair of random variables conditioned on a third variable. 1.1 Zero-infinity dichotomy Consider a typical situation from dynamical systems, where the random variable V_{t+1} is determined by U_{t},V_{t} via some deterministic map T , that is, V_{t+1}=T(U_{t},V_{t}). (2) If V_{t+1} does not depend on U_{t} , that is, V_{t+1}=T_{0}(V_{t}) , then we trivially have zero information flow T_{U\to V,t}=0 . In terms of probability distributions, this case corresponds to the regular conditional probability \mathbb{P}(V_{t+1}\in\cdot|V_{t}=v_{t})=\delta_{T_{0}(v_{t})} being a dirac delta. Otherwise, one expects T_{U\to V,t}>0 to quantify the amount of information flowing from U to V at time t . For example, if the map T is highly “ambiguous”, then the knowledge about the states of U_{t},V_{t} does not resolve much uncertainty about the state of V_{t+1} . Example 1.1 . Consider two maps T_{1}(u,v)=100(u+v)\mod 1 and T_{2}(u,v)=u+v\mod 1 . The knowledge about the states of U_{t},V_{t} up to 10^{-2} precision is completely lost via T_{1} and trivially informs that V_{t+1}=T_{1}(U_{t},V_{t}) lies in [0,1] , whereas this knowledge under T_{2} informs about the state of V_{t+1}=T_{2}(U_{t},V_{t}) up to precision 2\times 10^{-2} . Therefore, we may expect T_{U\to V,t} to be smaller in the more ambiguous case of V_{t+1}=T_{1}(U_{t},V_{t}) than in the case of V_{t+1}=T_{2}(U_{t},V_{t}) . However, under some mild continuity assumptions on the distribution of V_{t+1} , we see that in both cases, T_{U\to V,t}=\infty . This holds more generally for any measurable map T . Throughout this paper, we assume that the random variables take values in standard measurable spaces, unless otherwise stated. This implies the existence and essential uniqueness of regular conditional probabilities and disintegrations; for details see Appendix A . Theorem A (infinite information flow): Assume that for a positive measure set of outcomes v_{t} of V_{t} , the regular conditional probability distribution \mathbb{P}(V_{t+1}\in\cdot|V_{t}=v_{t}) of V_{t+1} in Eq. ( 2 ) charges an atomless continuum. Then, the transfer entropy T_{U\to V,t} from U to V at time t is infinite. Remark 1.2 . The positive measure set is with respect to the distribution of V_{t} . We say that a probability measure \mu charges an atomless continuum if there is a measurable set B such that \mu(B)>0 and \mu(\{b\})=0 for each point b\in B . The assumption of Theorem A says that V_{t} alone does not fully determine V_{t+1} but rather leaves a rich continuum of possible values for V_{t+1} . This is the case, for example, when V_{t+1}=T_{1}(U_{t},V_{t}) or V_{t+1}=T_{2}(U_{t},V_{t}) as in Example 1.1 with V_{t} and U_{t} independent and following the uniform distribution on [0,1] . Theorem 3.7 gives an equivalent but slightly different formulation of Theorem A and is proven in Section 3.3 . The zero-infinity dichotomy of T_{U\to V,t} gives a yes-or-no answer to the question of information flow. A key step in the proof of Theorem A is to disintegrate the conditional mutual information into mutual information between conditioned variables. We believe that this result is interesting in its own right and state it below. Theorem B (disintegration of conditional mutual information): The conditional mutual information I(X;Y|Z) of three random variables X,Y,Z is the average of the mutual information I(X_{z};Y_{z}) between conditioned versions X_{z},Y_{z} of X,Y defined in Eq. ( 5 ), that is, I(X;Y|Z)=\int I(X_{z};Y_{z})\mathrm{d}P_{Z}(z). Figure 1: Disintegrated distributions. The main histogram at the bottom illustrates the distribution P_{Z} of variable Z , which, together with Y , determines X=T(Y,Z) via a measurable map T . The joint distribution P_{XYZ} disintegrates into (P_{XYZ})_{z} for each realization of Z=z , which can be interpreted as the joint distribution P_{X_{z}Y_{z}} of

Read the complete text