A Generalized Information Bottleneck Theory of Deep Learning
Charles Westphal, Stephen Hailes, & Mirco Musolesi · 2025
Read A Generalized Information Bottleneck Theory of Deep Learning by Charles Westphal, Stephen Hailes, & Mirco Musolesi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-26327.txt.
Introduction
Charles Westphal and co-authors rebuilt in 2025 a leading theory of how neural networks learn, fixing its fuzzy math and its unreliable measurement. The information bottleneck idea says networks squeeze inputs into compact codes that keep what matters. In practice the squeeze was hard to define and harder to measure.
Their generalized version runs on synergy: information that only appears when features are processed jointly. A computable synergy score replaces the old ideal quantity, and the classic objective provably sits beneath the new one. Across many architectures, including ones where the old theory fails, training showed the expected compression phases. The dynamics also read cleanly in both image and language models. The authors tie the view to robustness against attacks.
Catalog record
| CREATOR | Charles Westphal, Stephen Hailes, & Mirco Musolesi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-26327
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Generalized Information Bottleneck Theory of Deep Learning Charles Westphal, Stephen Hailes, & Mirco Musolesi Abstract The Information Bottleneck (IB) principle offers a compelling theoretical framework to understand how neural networks (NNs) learn. However, its practical utility has been constrained by unresolved theoretical ambiguities and significant challenges in accurate estimation. In this paper, we present a Generalized Information Bottleneck (GIB) framework that reformulates the original IB principle through the lens of synergy, i.e., the information obtainable only through joint processing of features. We provide theoretical and empirical evidence demonstrating that synergistic functions achieve superior generalization compared to their non-synergistic counterparts. Building on these foundations we re-formulate the IB using a computable definition of synergy based on the average interaction information (II) of each feature with those remaining. We demonstrate that the original IB objective is upper bounded by our GIB in the case of perfect estimation, ensuring compatibility with existing IB theory while addressing its limitations. Our experimental results demonstrate that GIB consistently exhibits compression phases across a wide range of architectures (including those with ReLU activations where the standard IB fails), while yielding interpretable dynamics in both CNNs and Transformers and aligning more closely with our understanding of adversarial robustness. 1 Introduction Deep learning has achieved remarkable practical success, yet our theoretical understanding of how neural networks learn effective representations remains incomplete ( Shwartz-Ziv & Tishby 2017 ) . Information theory offers a principled framework for analyzing deep learning, as information-theoretic quantities are invariant to invertible transformations and provide interpretable units of measurement ( Cover & Thomas 1991 ) . The Information Bottleneck (IB) principle, introduced by Tishby et al. 1999 , has emerged as a particularly influential framework for understanding neural network learning dynamics, providing Figure 1: This schematic illustrates information plane dynamics during training, with trajectories color-coded from early epochs (light colors) to late epochs (dark purple), showing distinct fitting and compression phases. insights into diverse phenomena including adversarial robustness ( Ma et al. 2021 ) , the effects of dropout regularization ( Achille & Soatto 2018 ) , and generalization bounds ( Kawaguchi et al. 2023 ) . Through this interpretation, the activations of our network can be viewed as a hidden state representation \mathcal{T} , which converges to a set of statistics captured by two competing terms. The first term, referred to as the prediction term , quantifies the mutual information (MI) between the hidden representation \mathcal{T} and the target Y , denoted as I(Y;\mathcal{T}) 1 1 1 For the mathematical notation used throughout this paper, see Appendix A . . It is straightforward to see that achieving training (and consequently test) accuracy above random guessing requires a network whose learned representation is well aligned with that of the target data. However, it is well established that optimizing solely for prediction accuracy can lead to overfitting. Consequently, the IB framework introduces a second term: the complexity term I(\mathcal{X};\mathcal{T}) , which quantifies the mutual information between the input data \mathcal{X} and the hidden representation. Optimizing this less intuitive term can be interpreted as an effort to minimize redundant and irrelevant information from the input that is encoded in the latent space. Overall, the IB framework posits that deep neural networks learn by solving the following Lagrangian optimization problem: \mathcal{L}_{\text{IB}}=\max_{p(\mathcal{T}|\mathcal{X})}\left[\underbrace{{\color[rgb]{0,0,1}I(\mathcal{T};Y)}}_{{\color[rgb]{0,0,1}\text{prediction term}}}-\underbrace{{\color[rgb]{1,0,0}\beta^{-1}I(\mathcal{X};\mathcal{T})}}_{{\color[rgb]{1,0,0}\text{complexity term}}}\right] (1) Shwartz-Ziv & Tishby 2017 suggested that the success of deep learning can be attributed to the ability of NNs to perform the aforementioned optimization problem in two distinct phases. First, a fitting phase , where both of the two introduced terms increase, and second a compression phase , where the complexity term decreases in size (refer to Figure 1 for a visual illustration of these processes). It was argued that this second compression phase was unique to deep models and helped explain their generalizability. In some cases, the flow of information through the latent space has been shown to align precisely with the IB’s theoretical predictions ( Shwartz-Ziv & Tishby 2017 ) . While the IB framework initially seemed to provide a complete explanation of how neural networks balance compression and predictive accuracy, Saxe et al. 2019 presented counterexamples that challenge this view. In particular, the authors showed that compression phases depend critically on the choice of activation function: networks with tanh activations exhibited compression across all layers, whereas ReLU -based networks did not. Despite the absence of a compression phase, the ReLU networks still generalized well. According to Goldfeld et al. 2019 , this occurs because the complexity term in deterministic networks is theoretically constant or infinite, rendering compression impossible. Consequently, the compression observed in tanh networks was not a genuine information-theoretic effect, but rather the result of injected randomness ( Saxe et al. 2019 ; Shwartz-Ziv & Tishby 2017 ; Geiger 2022 ) . In this paper, we address these issues by introducing a generalized formulation of the IB framework that is grounded in synergy. Synergy, a concept from multivariate information theory, captures the extra predictive power that arises when inputs are considered together rather than in isolation ( Williams & Beer 2010 ) . To motivate this perspective, we begin by asking: why synergy? We then present both theoretical arguments and empirical results showing that synergistic functions lead to improved generalization. Having established that synergistic functions generalize better, we construct the GIB by reformulating the IB through the lens of synergy. First, we introduce a point-wise mutual information (PMI)-based reweighting scheme that ensures we measure synergy specifically for correct predictions rather than arbitrary outputs. We then combine this reweighting with our feature-wise synergy decomposition, which uses the interaction information (II) to quantify information available only through joint processing of all features. Finally, we cast this as a Lagrangian optimization problem, yielding our GIB objective that measures how synergistically the input features combine to describe correct outputs. After deriving the GIB, we prove that, under perfect estimation, it can be lower bounded by the IB. Importantly, our formulation overcomes key theoretical limitations of standard IB, including the issue of infinite complexity terms. We demonstrate that GIB exhibits clear compression phases and interpretable learning dynamics across a wide range of scenarios where standard IB fails. In Figure 2 , we revisit the experiments presented in Saxe et al. 2019 and show that the GIB displays compression phases for five different activation functions while the IB is limited to one 2 2 2 Reading Information Planes Plots. Throughout, we visualize information dynamics using information plane plots. In these plots, the x -axis represents the complexity term and the y -axis represents the prediction term. For standard IB, these are I(\mathcal{X};\mathcal{T}) and I(\mathcal{T};Y) , respectively. For the full formulation of the GIB, see Section 3 . Trajectories are color-coded by training epoch, progressing from early training (dark pink/blue) to late training (light green/yellow). Blue trajectories correspond to the standard IB dynamics, whereas pink trajectories depict the dynamics under our GIB formulation. Movement leftward indicates compression (reduction of redundant information), while movement upward indicates improved prediction. For the IB we only report the information plane of the final layer as this is where compression dynamics are most readily observed. Meanwhile, the GIB is formulated based on inputs and therefore only produces one information plane per training.