Information Gradient for Directed Acyclic Graphs: A Score-based Framework for End-to-End Mutual Information Maximization

Tadashi Wadayama · 2026

Read Information Gradient for Directed Acyclic Graphs: A Score-based Framework for End-to-End Mutual Information Maximization by Tadashi Wadayama (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01789.txt.

Introduction

Communication and sensing systems can be drawn as directed networks where signals flow from inputs to outputs through random internal stages. Tadashi Wadayama develops a general method for computing how mutual information, the amount the output reveals about the input, changes as any internal parameter is tuned.

The unified formula uses marginal and conditional score functions, and it can be evaluated with standard automatic differentiation tools through vector-Jacobian products. Tests on linear multipath networks and nonlinear channels confirm the estimator reproduces true gradients and can maximize information end to end under resource limits. Wadayama also extends the framework to calibrating digital twins by minimizing Fisher divergence, broadening its use beyond communications design.

Catalog record

CREATORTadashi Wadayama
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-01789

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Information Gradient for Directed Acyclic Graphs: A Score-based Framework for End-to-End Mutual Information Maximization Tadashi Wadayama Abstract This paper presents a general framework for end-to-end mutual information maximization in communication and sensing systems represented by stochastic directed acyclic graphs (DAGs). We derive a unified formula for the (mutual) information gradient with respect to arbitrary internal parameters, utilizing marginal and conditional score functions. We demonstrate that this gradient can be efficiently computed using vector-Jacobian products (VJP) within standard automatic differentiation frameworks, enabling the optimization of complex networks under global resource constraints. Numerical experiments on both linear multipath DAGs and nonlinear channels validate the proposed framework; the results confirm that the estimator, utilizing score functions learned via denoising score matching, accurately reproduces ground-truth gradients and successfully maximizes end-to-end mutual information. Beyond maximization, we extend our score-based framework to a novel unsupervised paradigm: digital twin calibration via Fisher divergence minimization. I Introduction Designing complex communication and sensing systems to maximize information flow is a central challenge in modern engineering. From next-generation wireless networks aiming for semantic communication [ 11 ] to distributed sensor fusion in IoT, the paradigm is shifting from optimizing individual modules to end-to-end system design. In these scenarios, mutual information (MI) [ 1 ] serves as the gold standard objective function, capturing nonlinear dependencies that simple correlations miss. However, maximizing MI in such high-dimensional, multi-stage systems presents a formidable computational barrier. Unlike deterministic loss functions (e.g., mean squared error) where gradients are easily computed via backpropagation, MI depends on the joint probability density, which evolves in complex ways through the system dynamics. Existing variational bounds focus primarily on estimating the MI value; utilizing them for gradient-based optimization of internal system parameters often leads to high variance and instability. Essentially, we lack a scalable mechanism to ”backpropagate” information measures through complex stochastic graphs. In a companion paper [ 2 ] , we took a first step toward addressing this challenge by introducing the information gradient framework. This theory establishes an exact identity linking the gradient of MI to score functions [ 1 , 10 ] —the gradients of log-likelihoods. By leveraging modern score matching techniques [ 3 , 4 , 5 ] , this framework allows for efficient gradient estimation without closed-form likelihoods. However, the scope of [ 2 ] was limited to single-stage channels of the form Y=f_{\bm{\eta}}(X)+Z . Real-world systems are rarely this simple; they involve intricate topologies with branching paths, merging signals, and multi-hop processing, effectively forming stochastic networks. Applying the basic formula to such architectures is non-trivial, as a single parameter can influence the final output through multiple interacting pathways, complicating the derivation of the score function and the associated chain rules. This paper generalizes the information gradient framework to arbitrary stochastic systems represented by directed acyclic graphs (DAGs) . We view the system as a computational graph where nodes represent stochastic variables and edges represent parameterized nonlineartransformations. This means that any stochastic network system implementable in modern automatic differentiation frameworks (e.g., PyTorch, TensorFlow, JAX) can now be optimized for MI using our score-based estimator. Our main contribution is a unified gradient formula applicable to any internal parameter within a DAG, regardless of its depth or topological complexity. Crucially, we bridge the gap between information theory and deep learning implementation: we show that the generalized information gradient can be efficiently computed using standard vector-Jacobian products (VJP) [ 6 ] . Moreover, we demonstrate that canonical multi-user models, such as multiple access channels (MAC) and broadcast channels (BC), are naturally represented as DAGs. Consequently, our method serves as a unified numerical solver for exploring the achievable rate regions of complex nonlinear channels where closed-form analytical characterizations are intractable. To rigorously validate this capability, we show that our score-based information gradient exactly reproduces the known capacity region of the Gaussian MAC. This generalization opens up broad practical applications. It enables the end-to-end optimization of transmitter and receiver neural networks over unknown nonlinear channels, a key requirement for semantic communications . Furthermore, we extend the framework to a novel unsupervised paradigm: digital twin calibration . By minimizing the Fisher divergence—a natural byproduct of our score-based formulation— we propose a method to align simulation parameters with physical reality using only output observations. II Preliminaries In this section, we introduce the basic notations and review key concepts regarding the fundamental information gradient identity established in our companion paper [ 2 ] . II-A Notations We use boldface lowercase letters (e.g., \bm{x} ) to denote vectors and boldface uppercase letters (e.g., \bm{A} ) for matrices. Random vectors are denoted by uppercase letters (e.g., X,Y ), and their realizations by lowercase letters (e.g., \bm{x},\bm{y} ). The probability density function (PDF) of a continuous random vector X is denoted by p_{X}(\bm{x}) . The expectation operator is denoted by \mathbb{E}[\cdot] . We use h(X)\equiv-\mathbb{E}[\log p_{X}(X)] to denote the differential entropy of X , and I(X;Y)\equiv h(Y)-h(Y|X) for the mutual information between X and Y [ 1 ] . The Jacobian matrix of a vector-valued differentiable function \bm{f}(\bm{x}) with respect to \bm{x} is denoted by D_{\bm{x}}\bm{f}\equiv\frac{\partial\bm{f}}{\partial\bm{x}} . II-B Score Functions The gradient of the log-likelihood with respect to the data variable is termed the score function [ 1 ] . For a random vector Y\in\mathbb{R}^{d} with PDF p_{Y}(\bm{y}) , the (marginal) score function is defined as \bm{s}_{Y}(\bm{y})\equiv\nabla_{\bm{y}}\log p_{Y}(\bm{y}). (1) Similarly, the conditional score function for Y given X is \bm{s}_{Y|X}(\bm{y}|\bm{x})\equiv\nabla_{\bm{y}}\log p_{Y|X}(\bm{y}|\bm{x}) . A fundamental property of the score function is that its expectation under the corresponding distribution is zero, provided that the PDF vanishes at the boundary: \displaystyle\mathbb{E}[\bm{s}_{Y}(Y)] \displaystyle=\int p_{Y}(\bm{y})\nabla_{\bm{y}}\log p_{Y}(\bm{y})d\bm{y} \displaystyle=\int\nabla_{\bm{y}}p_{Y}(\bm{y})d\bm{y}=\bm{0}. (2) This property plays a crucial role in simplifying gradient derivations. II-C Related works Our framework including [ 9 ] is grounded in the deep correspondence between information theory and thermodynamics, specifically the interpretation of noise injection as a diffusion process. This perspective originates from Stam [ 14 ] , who utilized the de Bruijn identity—relating the derivative of entropy to Fisher information—to establish the entropy power inequality. Building on this diffusion-based view, Brown [ 15 ] and Barron [ 16 ] developed the entropic central limit theorem, proving the monotonic convergence of Fisher information. In the context of Gaussian channels, Guo, Shamai, and Verdú [ 17 ] unified these concepts through the I-MMSE relationship, identifying the derivative of mutual information with the minimum mean-square error. While these classical studies primarily employed these relationships for theoretical proofs of inequalities, our work repurposes them for practical computation. We extend the utility of these fundamental identities from theoretical analysis to data-driven optimization. By generalizing these connections to arbitrary stochastic DAGs and estimating the requisite score functions from data, we transform these classical thermodynamic links into a scalable engine for end-to-end mutual information maximization. II-D Information Gradient Identity In our companion paper [ 2 ] , we derived a fundamental identity that relates the gradient of mutual information to score functions. Consider a general parameterized system where the output is given by Y=f_{\bm{\eta}}(X)+Z , with input X\in\mathbb{R}^{n} , where f_{\bm{\eta}}:\mathbb{R}^{n}\to\mathbb{R}^{m} is a differentiable function parameterized by \bm{\eta}\in\mathbb{R}^{d} , and Z\in\mathbb{R}^{m} is an additive noise vector. Under standard regularity conditions, the gradient of the mutual information with respect to \bm{\eta} can be expressed as \nabla_{\bm{\eta}}I(X;Y)=\mathbb{E}\left[(D_{\bm{\eta}}Y)^{\top}(\bm{s}_{Y|X}(Y|X)-\bm{s}_{Y}(Y))\right]. (3) This identity states

Read the complete text