Structured IB: Improving Information Bottleneck with Structured Feature Learning

Hanzhe Yang, Youlong Wu, Dingzhu We, Yong Zhou, Yuanming Shi · 2024

Read Structured IB: Improving Information Bottleneck with Structured Feature Learning by Hanzhe Yang, Youlong Wu, Dingzhu We, Yong Zhou, Yuanming Shi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08222.txt.

Introduction

Hanzhe Yang and co-authors improve the information bottleneck method with structured feature learning in a 2024 paper. The bottleneck principle squeezes a network's representation to keep only what a task needs, aiding generalization and robustness. The common Lagrangian version depends heavily on design quality, which is error prone. Their framework adds auxiliary encoders that hunt for informative features the main encoder missed.

Those extra features enrich the representation before the squeeze. In experiments, the structured version beat the original method on prediction accuracy and on keeping task relevant information. It did so even with a smaller network. The authors frame the gain as better raw material: a bottleneck can only preserve what the features first capture.

Catalog record

CREATORHanzhe Yang, Youlong Wu, Dingzhu We, Yong Zhou, Yuanming Shi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08222

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structured IB: Improving Information Bottleneck with Structured Feature Learning Hanzhe Yang, Youlong Wu, Dingzhu We, Yong Zhou, Yuanming Shi Abstract The Information Bottleneck (IB) principle has emerged as a promising approach for enhancing the generalization, robustness, and interpretability of deep neural networks, demonstrating efficacy across image segmentation, document clustering, and semantic communication. Among IB implementations, the IB Lagrangian method, employing Lagrangian multipliers, is widely adopted. While numerous methods for the optimizations of IB Lagrangian based on variational bounds and neural estimators are feasible, their performance is highly dependent on the quality of their design, which is inherently prone to errors. To address this limitation, we introduce Structured IB, a framework for investigating potential structured features. By incorporating auxiliary encoders to extract missing informative features, we generate more informative representations. Our experiments demonstrate superior prediction accuracy and task-relevant information preservation compared to the original IB Lagrangian method, even with reduced network size. Code — https://github.com/HanzheYang19/Structure-IB Introduction The Information Bottleneck (IB) principle, a representation learning approach rooted in information theory, was introduced by ( Tishby, Pereira, and Bialek 2000 ) . The core idea of IB is to extract a feature, denoted as Z , that captures relevant information about the target Y from the input X . Specifically, IB aims to maximize the mutual information (MI) between Z and Y while constraining the MI between X and Z , effectively preserving only information essential for predicting Y . Mathematically, this is formulated as: \max_{p(Z|X)}I(Z,Y)\;\;\;\text{s.t.}\;\;I(X,Z)<r, (1) where r controls the level of compression. Solving this constrained optimization problem directly is challenging. To address this, IB Lagrangian methods (see, ( Tishby, Pereira, and Bialek 2000 ; Yu et al. 2024 ; Rodríguez Gálvez, Thobaben, and Skoglund 2020 ; Shamir, Sabato, and Tishby 2010 ) ) introduce a positive Lagrange multiplier \beta , transforming the problem into: \min_{p(Z|X)}-I(Z,Y)+\beta I(X,Z), (2) where \beta is typically within the range [0, 1] ( Rodríguez Gálvez, Thobaben, and Skoglund 2020 ) . IB and its variations ( Alemi et al. 2016 ; Kolchinsky, Tracey, and Wolpert 2019 ; Rodríguez Gálvez, Thobaben, and Skoglund 2020 ) have been applied to various domains, including image segmentation ( Bardera et al. 2009 ) , document clustering ( Slonim and Tishby 2000 ) , and semantic communication ( Yang et al. 2023 ; Xie et al. 2023 ) . Recent research has linked IB to Deep Neural Networks (DNNs) in supervised learning, where X represents input features, Y is the target (e.g., class labels), and Z corresponds to intermediate latent representations. Studies ( Shwartz-Ziv and Tishby 2017 ; Lorenzen, Igel, and Nielsen 2021 ) have shown that IB can explain certain DNN training dynamics. Additionally, IB has been shown to improve DNN generalization ( Wang et al. 2021 ) and adversarial robustness ( Alemi et al. 2016 ) . Despite their practical success, IB methods face computational challenges during optimization. Variational bounds and neural estimators ( Belghazi et al. 2018b ; Pichler et al. 2022 ) have been proposed to mitigate these issues ( Alemi et al. 2016 ; Kolchinsky, Tracey, and Wolpert 2019 ; Rodríguez Gálvez, Thobaben, and Skoglund 2020 ) . However, the approximations inherent in these methods can deviate from the original objective due to the nature of upper or lower bounds and estimators. This paper introduces the Structured Information Bottleneck (SIB) framework for IB Lagrangian methods. Unlike traditional IB, which uses a single feature, SIB explores structured features. We divide the feature extractor into a main encoder and multiple auxiliary encoders. The main encoder is trained using the IB Lagrangian, while auxiliary encoders aim to capture information missed by the main encoder. These features are combined to form the final feature. Our experiments demonstrate that SIB achieves higher I(Z,Y) for the same compression level I(X,Z) , improves prediction accuracy, and is more parameter-efficient. We also analyze how auxiliary features enhance the main feature. The contributions of our work can be summarized as follows: • Proposed a novel SIB framework for IB Lagrangian methods , departing from the traditional single feature approach by incorporating multiple auxiliary encoders. • Developed a novel training methodology for SIB , involving a two-stage process: initial training of the main encoder using the IB Lagrangian, followed by training auxiliary encoders to capture complementary information while maintaining distinctiveness from the main feature. • Demonstrated superior performance of SIB compared to existing methods in terms of achieving higher mutual information between the compressed representation and the target variable for the same level of compression, as well as improved prediction accuracy and parameter efficiency. • Provided a comprehensive analysis of how auxiliary features enhance the main feature , shedding light on the underlying mechanisms of the proposed framework. Relative Work The IB Lagrangian has been extensively studied in representation learning. Several methods have been proposed to optimize it using DNNs. ( Alemi et al. 2016 ) introduced Variational Information Bottleneck (VIB), deriving variational approximations for the IB Lagrangian: I(X,Z)\leq\mathbb{E}_{q(x,z)}\log q(z|x)-\mathbb{E}_{q(t)}\log v(t), (3) I(Z,Y)\geq\mathbb{E}_{p(y,z)}\log q(y|z)+H(Y), (4) where q(\cdot|\cdot) and p(\cdot|\cdot) represent the probabilistic mapping and variational probabilistic mapping respectively, q(\cdot) is the probabilistic distribution, H(\cdot) is the entropy, and v(\cdot) is some prior distribution. Departing from VIB, Nonlinear Information Bottleneck (NIB) by Kolchinsky, Tracey, and Wolpert 2019 employs kernel density estimation ( Kolchinsky and Tracey 2017 ) to bound I(X,Z) : I(X,Z)\leq-\frac{1}{N}\sum_{i=1}^{N}\log\frac{1}{N}\sum_{j=1}^{N}e^{-D_{KL}[q(z|x_{i})\|q(z|x_{j})]}, (5) where D_{KL}(\cdot\|\cdot) is the KL divergence and N is the total number of samples in the dataset. However, Rodriguez Galvez 2019 demonstrated that optimizing the IB Lagrangian for different values of \beta cannot explore the IB curve when Y is a deterministic function of X . To address this, they proposed the Square IB (sqIB) by simply using the square of compression term (I(X,Z))^{2} instead of I(X,Z) in the Lagrangian function. Furthermore, Rodríguez Gálvez, Thobaben, and Skoglund 2020 showed that applying any monotonically increasing and strictly convex functions on I(X,Z) is able to explore the IB curve. Additionally, the authors of Yu et al. 2024 extended the IB for regression tasks with Cauchy-Schwarz divergence. There are also other approaches that focus on neural estimators for directly approximating MI, entropy, and differential entropy (see, ( Belghazi et al. 2018b ; Pichler et al. 2022 ) ), enabling direct IB Lagrangian optimization without variational bounds. While these methods offer theoretical advancements, their implementations can be error-prone due to specific implementation details. Our proposed SIB framework introduces auxiliary encoders to capture complementary information and a novel training process. Unlike prior work, we leverage the IB Lagrangian for both main and auxiliary encoder training and include an additional term to prevent feature overlap. Methodology In this section, we first present the architecture of the proposed network. Subsequently, the training process will be illustrated in detail. Network Architecture Our network architecture comprises three key components: the main encoder, auxiliary encoders, and the decoder, as illustrated in Figure 1 . The main encoder extracts the primary feature from the input, while auxiliary encoders capture complementary features to enrich the information content of the primary feature. These features are combined using a weighted aggregation function f(\cdot) to produce a comprehensive feature representation. In this work, we employ a simple weighted summation, which will be justified in the following subsection. The decoder processes

Read the complete text