Structural Entropy Guided Probabilistic Coding
Xiang Huang, Hao Peng, Li Sun, Hui Lin, Chunyang Liu, Jiang Cao, Philip S. Yu · 2024
Read Structural Entropy Guided Probabilistic Coding by Xiang Huang, Hao Peng, Li Sun, Hui Lin, Chunyang Liu, Jiang Cao, Philip S. Yu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08841.txt.
Introduction
Xiang Huang and six co-authors add structure awareness to probabilistic embeddings. Their paper appeared in two thousand twenty-four. Probabilistic codes map each data point to a distribution, not a point. That spread captures uncertainty in the data. Most methods tune each latent variable on its own. Links between the variables go unused. The information bottleneck principle guides the usual trade-offs.
Their model, SEPC, adds a structural entropy term to the loss. Latent relations now shape the coding directly. A probabilistic encoding tree adapts the idea to regression tasks. Regression is recast as classification along the tree. Structure guides both compression and accuracy. Representation learning gains a way to respect its own wiring.
Catalog record
| CREATOR | Xiang Huang, Hao Peng, Li Sun, Hui Lin, Chunyang Liu, Jiang Cao, Philip S. Yu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08841
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Structural Entropy Guided Probabilistic Coding Xiang Huang, Hao Peng, Li Sun, Hui Lin, Chunyang Liu, Jiang Cao, Philip S. Yu Abstract Probabilistic embeddings have several advantages over deterministic embeddings as they map each data point to a distribution, which better describes the uncertainty and complexity of data. Many works focus on adjusting the distribution constraint under the Information Bottleneck (IB) principle to enhance representation learning. However, these proposed regularization terms only consider the constraint of each latent variable, omitting the structural information between latent variables. In this paper, we propose a novel structural entropy-guided probabilistic coding model, named SEPC. Specifically, we incorporate the relationship between latent variables into the optimization by proposing a structural entropy regularization loss. Besides, as traditional structural information theory is not well-suited for regression tasks, we propose a probabilistic encoding tree, transferring regression tasks to classification tasks while diminishing the influence of the transformation. Experimental results across 12 natural language understanding tasks, including both classification and regression tasks, demonstrate the superior performance of SEPC compared to other state-of-the-art models in terms of effectiveness, generalization capability, and robustness to label noise. The codes and datasets are available at https://github.com/SELGroup/SEPC . 1 Beihang University, 2 Guangdong Laboratory of Artificial Intelligence and Digital Economy 3 North China Electric Power University 4 China Academic of Electronics and Information Technology 5 Didi Chuxing 6 Academy of Military Sciences 7 University of Illinois Chicago {huang.xiang, penghao}@buaa.edu.cn, [email protected], [email protected] [email protected], [email protected], [email protected] Introduction Probabilistic embedding ( Vilnis and McCallum 2015 ) is a flexible representation learning method aiming to learn the underlying probability distribution of data. It has been broadly applied to various domains such as graph structural learning ( Sun et al. 2022 ) , computer vision ( Kim et al. 2021 ; Oh et al. 2019 ; Shi and Jain 2019 ; Fischer 2020 ) , and natural language processing ( Mahabadi, Belinkov, and Henderson 2021 ; Hu et al. 2024 ; Hu et al. 2022 ) . In contrast to deterministic embedding ( Dong, Yan, and Wang 2024 ; Xu et al. 2024 ) , which maps the input into a fixed latent variable representation, probabilistic embedding represents each data point as a probability distribution. Hence, probabilistic embedding inherently accounts for the uncertainty and complexity of data by controlling the spread of the probability density over the learning latent space ( Oh et al. 2019 ) , showcasing better discriminative ability and robustness. The mainstream probabilistic embedding methods are grounded in the Information Bottleneck (IB) principle ( Tishby, Pereira, and Bialek 2000 ; Tishby and Zaslavsky 2015 ) . IB aims to find compressed representations that maintain as much information as possible for the prediction task while removing as much irrelevant information as possible. Specifically, it seeks the latent representation Z that is maximally informative about the target Y (i.e., maximize mutual information I(Y;Z) ) while being minimally informative about the input data X (i.e., minimize mutual information I(X;Z) ) ( Sun et al. 2022 ) . The former target is typically achieved with common task losses like cross entropy (CE) loss or mean squared error (MSE) loss, whereas various regularization losses are proposed for the latter goal. VIB ( Alemi et al. 2017 ) assumes the prior distribution of Z is the standard normal distribution and utilizes Kullback–Leibler (KL) divergence to regularize the learning distribution p(z|x) . Sparse IB ( Chalk, Marre, and Tkacik 2016 ) changes the prior distribution of VIB to the Student-t distribution to achieve relevant and sparse coding. MEIB ( An, Jammalamadaka, and Chong 2023 ) lifts the prior distribution constraint of VIB and instead uses maximum conditional entropy H(Z|X) as the only regularization. SPC ( Hu et al. 2024 ) omits the decoder of VIB and proposes an additional structured regularization that encourages class-level uniformity within the latent space under the multivariate Gaussian distribution. However, all of them focus solely on the individual latent variable Z or the constraint of Z with the label Y , neglecting the structural information between latent variables. In recent years, structural entropy theory ( Li and Pan 2016 ) has demonstrated its advantage in capturing hierarchical structural information and has been widely used in various fields like node classification ( Duan et al. 2024 ) , graph structural learning ( Zou et al. 2023 ) , and contrastive learning ( Wu et al. 2023 ) . It considers the structural information of the original inputs by modeling the input data as a graph and then converting the graph into an encoding tree. The data points are the leaf nodes of the encoding tree, and each upper node represents a partition, resulting in a hierarchical clustering of the input data. Low-depth tree nodes depict more coarse-grained clusters of the input data. Each node in the encoding tree has its own structural entropy. The structural entropy of the encoding tree is calculated by summing the structural entropy of all non-root nodes, representing the overall structural information of the input. Previous work ( Wang et al. 2023 ; Zeng, Peng, and Li 2023 ) mostly focuses on minimizing the structural entropy of the encoding trees to obtain the optimized encoding tree or embeddings of input data, aiming at learning as much task-related information as possible. However, the potential for using structural entropy for regularization remains underexplored. Additionally, as the structural entropy is designed for classification tasks, how to effectively leverage it in the regression task is still a problem. In this paper, we propose a structural entropy-guided probabilistic coding model, named SEPC. We present a structural entropy-based regularization loss that incorporates structural information between latent variables. Specifically, we first construct the adjacency matrix based on the similarity between embeddings of latent variables and propose to maximize the structural entropy of the induced graph, which helps improve the generalization of the model by separating the probabilistic distribution of each latent variable. Additionally, we design a probabilistic encoding tree to adapt our structural entropy loss in regression tasks. We first discretize and soften regression labels into soft classification labels (i.e., each data point belongs to multiple classes with varying probabilities), diminishing the influence of unsuitable classification caused by using only discretization ( Pintea et al. 2023 ) . To adapt structural entropy to such soft labels, we relax the constraint that one child belongs to one parent in the encoding tree, allowing each child to connect to all upper-level nodes with varying probabilities. Extensive experiments are conducted on 12 natural language understanding tasks, including 10 classification tasks and 2 regression tasks. Comparative results and analysis demonstrate that the proposed SEPC enjoys superior effectiveness, generalization, and robustness compared to the state-of-the-art (SOTA) baselines. The main contributions are summarized as follows: • We present a structural entropy based regularization loss, incorporating the structural information between data points into model regularization. To our knowledge, this is the first time that maximizing structural entropy has been utilized as a regularization loss. • We propose a probabilistic encoding tree for soft classification labels and present an effective method to utilize structural entropy for regression tasks for the first time. • Extensive experiments on 12 datasets demonstrate that SEPC achieves SOTA performance in classification and regression tasks regarding effectiveness,