A Hierarchical Decomposition of Kullback-Leibler Divergence: Disentangling Marginal Mismatches from Statistical Dependencies

William Cook · 2025

Read A Hierarchical Decomposition of Kullback-Leibler Divergence: Disentangling Marginal Mismatches from Statistical Dependencies by William Cook (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09029.txt.

Introduction

William Cook takes apart the Kullback-Leibler divergence, the standard yardstick for how far one probability distribution sits from another. In many variables its single number hides its own causes: each variable may drift on its own, or the variables may lean on each other. Cook's decomposition separates the two sources exactly.

In his 2025 paper, total divergence against an independent reference splits into the sum of each variable's own mismatch plus the total correlation among them. A Möbius inversion over subsets then unpacks that correlation into signed pairwise, triple, and higher-order parts, all in plain Shannon terms. The result is a full ledger: any divergence can now be billed to the margins or to each layer of dependence.

Catalog record

CREATORWilliam Cook
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-09029

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Hierarchical Decomposition of Kullback-Leibler Divergence: Disentangling Marginal Mismatches from Statistical Dependencies William Cook Abstract The Kullback-Leibler (KL) divergence is a foundational measure for comparing probability distributions, yet in multivariate settings, its structure is often opaque—conflating marginal mismatches and statistical dependencies. We derive an algebraically exact, additive, and hierarchical decomposition of the KL divergence ( KL ( P k ∥ Q ( ⊗ k ) ) \mathrm{KL}(P_{k}\|Q^{(\otimes k)}) ) between a joint distribution ( P k P_{k} ) and a product reference ( Q ( ⊗ k ) Q^{(\otimes k)} ). The total divergence splits into the sum of marginal KLs, ( ∑ i = 1 k KL ( P i ∥ Q ) \sum_{i=1}^{k}\mathrm{KL}(P_{i}\|Q) ), and the total correlation ( C ⁡ ( P k ) C(P_{k}) ), which we further decompose as ( C ⁡ ( P k ) = ∑ r = 2 k I ( r ) ​ ( P k ) C(P_{k})=\sum_{r=2}^{k}I^{(r)}(P_{k}) ) using Möbius inversion on the subset lattice. Each ( I ( r ) I^{(r)} ) quantifies the distinct contribution of ( r r )-way statistical interactions to the total divergence. This yields a decomposition of this form that is both algebraically complete and interpretable using only standard Shannon quantities, with no approximations or model assumptions. Numerical validation using hypergeometric sampling confirms exactness to machine precision across diverse system configurations. This framework enables precise diagnosis of divergence origins—marginal vs. interaction—across applications in machine learning, econometrics, and complex systems. 1 Introduction Understanding why high-dimensional probability distributions differ—beyond merely quantifying how much—is a central challenge in statistics and machine learning. The Kullback-Leibler (KL) divergence, or relative entropy [ 1 ] , provides a foundational measure of the distinction between a ‘true’ distribution P and a model or reference distribution Q . Its applications are widespread, spanning machine learning (e.g., variational inference, model selection) [ 2 ] , statistical physics, and computational biology [ 3 ] . However, in multivariate systems described by a joint distribution P_{k}(X_{1},\dots,X_{k}) compared to a simpler reference such as a product distribution Q^{(\otimes k)}=\prod_{i=1}^{k}Q(X_{i}) , where each factor is identical to a fixed reference distribution Q , the total KL divergence \mathrm{KL}(P_{k}\|Q^{(\otimes k)}) yields a single scalar value. This aggregate measure conflates two distinct sources of divergence: mismatches between each individual marginal distribution P_{i} and the reference Q , and statistical dependencies (interactions) among the variables X_{1},\dots,X_{k} within P_{k} that are absent in the independent reference Q^{(\otimes k)} . Disentangling these components is crucial for interpreting the sources of divergence or model misfit. While established measures like mutual information and total correlation [ 4 ] quantify aspects of statistical dependency, they do not typically offer an exact decomposition of the KL divergence itself relative to Q^{(\otimes k)} . Nor do they systematically partition the divergence into contributions from marginal mismatches versus a hierarchy of interaction orders. This leaves key diagnostic questions unanswered: In machine learning, does a model’s poor fit (high KL divergence) stem primarily from misrepresenting individual feature distributions (marginal effects), or from failing to capture complex feature interactions (dependency effects)? In systems biology, does divergence from a baseline biological state arise from individual component dysregulation or from altered network interactions? This paper presents an exact, additive decomposition of \mathrm{KL}(P_{k}\|Q^{(\otimes k)}) that directly addresses this challenge. We demonstrate algebraically that: \mathrm{KL}(P_{k}\|Q^{(\otimes k)})=\underbrace{\left[\sum_{i=1}^{k}\mathrm{KL}(P_{i}\|Q)\right]}_{\text{Sum of Marginal Divergences}}+\underbrace{\left[\sum_{r=2}^{k}I^{(r)}(P_{k})\right]}_{\begin{subarray}{c}\text{Total Correlation /}\\ \text{Sum of Interactions ($r\geq 2$)}\end{subarray}} Here, the first term isolates the divergence due solely to differences between each marginal P_{i} and the reference Q . The second term is precisely the total correlation C(P_{k}) , which encapsulates the contributions from all statistical dependencies within P_{k} . Furthermore, we show this total correlation term decomposes hierarchically into the sum of r -way interaction information terms I^{(r)}(P_{k}) for r=2,\dots,k . The derivation leverages the fundamental properties of Shannon entropy and its relationship to interaction information, defined via Möbius inversion on the Boolean lattice of variable subsets [ 5 , 6 ] . This approach yields a principled and complete breakdown using only classical information-theoretic quantities. This decomposition provides a structured framework for dissecting KL divergence, offering fine-grained insights by cleanly separating marginal-level mismatches from the cumulative contributions of pairwise, triplet, and higher-order statistical interactions. It constitutes an exact, additive, and hierarchical decomposition of \mathrm{KL}(P_{k}\|Q^{(\otimes k)}) into these standard information-theoretic components. The framework is validated numerically using multivariate hypergeometric models, confirming its exactness. Preceding Section 6 is an accessible synthesis of the core contribution for readers without the necessary prior mathematical foundations implicitly assumed elsewhere. The paper is structured as follows: • Section 2 : Theoretical framework. Defines key concepts and rigorously derives the main decomposition theorem (Theorem 2.9 ). • Section 3 : Visual illustrations. Illustrates the decomposition using diagrams and empirical data from numerical cases. • Section 4 : Experiments and numerical validation. Details the numerical validation methodology and presents results confirming the decomposition’s exactness. • Section 5 : Intuition and Analogy. Accessible synthesis of the core contribution. • Section 6 : Discussion. Interprets the findings, discusses potential applications, connections to information geometry, and limitations. • Section 7 : Conclusion. Summarises the contribution and outlines future research directions. 2 Theoretical Framework 2.1 Definitions and Preliminaries Let X=(X_{1},\dots,X_{k}) be a discrete random vector over a finite alphabet \mathcal{X} , with joint probability distribution P_{k}(x_{1},\dots,x_{k}) . Let Q be a reference probability distribution on \mathcal{X} . Assumption 2.1 . We assume Q(x)>0 for all x in \mathcal{X} . This ensures that the logarithms in the KL divergence definitions are well-defined. Define the product reference distribution: Q^{(\otimes k)}(x_{1},\dots,x_{k})=\prod_{i=1}^{k}Q(x_{i}) We define Q as a fixed reference marginal distribution over the support \mathcal{X} . The product reference distribution is then constructed by taking the independent product of k identical marginals: Q^{(\otimes k)}(x_{1},\dots,x_{k}):=\prod_{i=1}^{k}Q(x_{i}). That is, we evaluate the divergence of P_{k} relative to an i.i.d. reference model in which each variable X_{i}\sim Q independently. For any subset S\subseteq[k]=\{1,\dots,k\} , let X_{S}=\{X_{i}:i\in S\} . Denote: • P_{S}(x_{S})=\sum_{x_{[k]\setminus S}}P_{k}(x_{1},\dots,x_{k}) , the marginal distribution of X_{S} . Notably, P_{i}(x_{i}) is the marginal distribution of X_{i} . • H(X_{S})=-\sum_{x_{S}}P_{S}(x_{S})\log_{2}P_{S}(x_{S}) , the Shannon entropy of X_{S} in bits, with the standard conventions H(X_{\emptyset})=0 and 0\log_{2}0=0 . Definition 2.2 (Interaction Information) . The interaction information for a non-empty subset S\subseteq[k] is defined via Möbius inversion of the negentropy function on the subset lattice: I(S)=-\sum_{T\subseteq S}(-1)^{|S|-|T|}H(X_{T}) This aligns with information geometry conventions [ 5 , 6 ] . Intuitively, I(S) measures the synergy or redundancy within the subset S —the portion of information generated by the variables acting together that cannot be attributed to any smaller sub-group. Remark 2.3 (Sign Convention Warning) . The interaction information I(S) is defined via Möbius inversion with alternating signs, following conventions from [ 5 ] . This implies that for subsets S with odd cardinality |S| , I(S) is typically negative, and for even |S| , positive. For instance: • I(\{i\})=-H(X_{i}) • I(\{i,j\})=I(X_{i};X_{j})\geq 0 (standard mutual information) • I(\{i,j,k\}) may be negative (synergy) or positive (redundancy). This is the negative of the quantity sometimes called co-information. This alternating-sign convention is crucial for the algebraic consistency in the decomposition (Theorem 2.9 ). It differs from conventions where all terms are defined non-negatively (e.g., [ 7 ] ). Remark 2.4 (Sign Convention and

Read the complete text