LH-Mix: Local Hierarchy Correlation Guided Mixup over Hierarchical Prompt Tuning
Fanshuang Kong, Richong Zhang, Ziqiao Wang · 2024
Read LH-Mix: Local Hierarchy Correlation Guided Mixup over Hierarchical Prompt Tuning by Fanshuang Kong, Richong Zhang, Ziqiao Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16963.txt.
Introduction
Sorting texts into a tree of labels, from broad topics down to narrow ones, is a standard task with a structural twist. Fanshuang Kong, Richong Zhang, and Ziqiao Wang improve a prompt tuning method for it in this 2024 paper. They build each text its own local slice of the label tree, rather than forcing one global tree on all texts.
That local slice is written into a depth-level prompt that captures parent-child links. A mixing technique then blends prompts to strengthen the hidden links among sibling labels, with the mix ratio guided by how strongly the local levels correlate. Across three widely used datasets, their model, LH-Mix, performed remarkably well. The authors credit the twin handling of both kinds of links.
Catalog record
| CREATOR | Fanshuang Kong, Richong Zhang, Ziqiao Wang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16963
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LH-Mix: Local Hierarchy Correlation Guided Mixup over Hierarchical Prompt Tuning Fanshuang Kong, Richong Zhang, Ziqiao Wang Abstract . Hierarchical text classification (HTC) aims to assign one or more labels in the hierarchy for each text. Many methods represent this structure as a global hierarchy, leading to redundant graph structures. To address this, incorporating a text-specific local hierarchy is essential. However, existing approaches often model this local hierarchy as a sequence, focusing on explicit parent-child relationships while ignoring implicit correlations among sibling/peer relationships. In this paper, we first integrate local hierarchies into a manual depth-level prompt to capture parent-child relationships. We then apply Mixup to this hierarchical prompt tuning scheme to improve the latent correlation within sibling/peer relationships. Notably, we propose a novel Mixup ratio guided by local hierarchy correlation to effectively capture intrinsic correlations. This L ocal H ierarchy Mix up (LH-Mix) model demonstrates remarkable performance across three widely-used datasets. Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (KDD ’25), August 3–7, 2025, Toronto, ON, Canada DOI: 10.1145/3690624.3709326 ISBN: 979-8-4007-1245-6/25/08 CCS: Computing methodologies Supervised learning by classification CCS: Computing methodologies Natural language processing 1. Introduction Figure 1. (a) A toy example of global hierarchy in HTC. (b) The local hierarchy of “CS/Machine Learning” and “Math/Statistics”, which are extracted from (a). (c) Transformation from explicit parent-child relations (a) to spatial inclusion relations in latent space. Mixup enables the capture of varying degrees of implicit sibling/peer label correlation through different Mixup ratios \lambda . Hierarchical Text Classification (HTC) is a variant of multi-label classification task characterized by labels organized in a predefined hierarchical structure ( Silla and Freitas, 2011 ) . This hierarchical structure captures relationships and dependencies among labels, with higher-depth labels containing lower-depth ones. Each text is then assigned to one or more labels within this hierarchy. The key challenge of HTC lies in effectively modeling the large-scale, imbalanced, and structured label hierarchy ( Mao et al., 2019b ) . Some existing works ( Deng et al., 2021 ; Zhou et al., 2020 ; Wang et al., 2022a ) consider the hierarchy as a directed acyclic graph, utilizing structure encoders to obtain label representations that incorporate hierarchical information. Nevertheless, this strategy allows each text to share the entire static global hierarchy, introducing redundancy in the graph ( Jiang et al., 2022 ) . This redundancy becomes particularly noticeable as the hierarchy size increases, with numerous labels becoming irrelevant to a specific given target text ( Im et al., 2023 ) . In contrast, alternative approaches proposed in ( Jiang et al., 2022 ; Im et al., 2023 ) invoke a local hierarchy, defined as a text-relevant sub-hierarchy extracted from the global hierarchy. Figure 1 (a-b) illustrate examples of these two types of hierarchies. This local hierarchy can be represented as an inclusion relationship in the latent space, as depicted in Figure 1 (c). For example, “Software” and “Machine Learning” should occupy the same subspace within “CS”, while “Geometry” and “Statistics” should exist in a subspace within the category of “Math”. On top of that, by treating a local hierarchy as a sequence, language models are able to capture parent-child relationships inherent in the hierarchy. Furthermore, motivated by the effectiveness of prompts in leveraging sequential information, in this work, we formulate the local hierarchy as a sequence using a hierarchical template to encode and align its structure. It is worth mentioning that labels may hold inherent relevance that extends beyond the constraints imposed by parent-child relationships in the hierarchy. Figure 1 (c) highlights a correlation between different local hierarchies. Specifically, while “CS/Machine Learning” and “Math/Statistics” occupy separate subspaces in the hierarchy, they may exhibit close proximity in the latent space, indicating a nuanced relationship that goes beyond the immediate hierarchical constraints. In fact, due to the widespread correlation among labels in HTC, including both sibling relationships and peer relationships ( Song et al., 2023 ) , capturing the correlation between local hierarchies becomes crucial. Existing methods, however, lack explicit mechanisms to tackle this specific challenge. An avenue of promise lies in Mixup ( Zhang et al., 2017 ) , a technique utilized for augmenting latent correlations between input pairs through the generation of intermediate samples ( Zhang et al., 2017 ) . Therefore, alongside hierarchical prompt tuning, we incorporate Mixup to reveal and leverage the implicit correlation between local hierarchies. More specifically, we first apply prompt tuning to HTC, where the local hierarchy is treated as a sequence. Virtual tokens are introduced to represent each depth in the hierarchy, thereby disassembling the parent-child relationship and aligning the hierarchy by depth. Within this hierarchical prompt tuning framework, we incorporate Mixup to capture implicit correlations among sibling and peer relationships. Notably, unlike Vanilla Mixup, which interpolates between the input and corresponding target using the same Mixup ratio sampled from a Beta Distribution, recent studies ( Sawhney et al., 2022 ; Zhang et al., 2022a ) adopt Mixup ratios based on instance correlations. This approach adjusts the mixture degree based on instance similarity, assigning higher mixture ratios to more similar instances to generate more informative samples. Conversely, less similar examples invoke a lower degree of mixture to prevent the generation of out-of-distribution data ( Baena et al., 2022 ) . As shown in Figure 1 (c), “CS/Machine Learning” should have a higher degree of mixing (e.g., with a Mixup ratio 0.6 ) with “Math/Statistic”, and a lower degree of mixing (e.g., with a Mixup ratio 0.9 ) to “Math/Geometry”. Motivated by this observation, we propose a novel approach for Mixup ratio control in our framework, dubbed local hierarchy Mixup (LH-Mix). In particular, LH-Mix assesses the correlation of a local hierarchy pair using the representation derived from the hierarchical prompt. By incorporating a heuristic function to determine the correlation and Mixup ratio, LH-Mix is able to learn hierarchical label correlations more effectively than vanilla Mixup. Below we summarize the main contributions of our work: • We introduce LH-Mix, which integrates Mixup based on a depth-level hierarchical prompt, effectively modeling the local hierarchy within sequences. To the best of our knowledge, LH-Mix is the first application of Mixup in HTC. • To capture the inherent correlations among local hierarchies, we propose a strategy for adjusting the Mixup ratio. This adaptive approach, guided by the local hierarchy correlation, encourages the generation of enriched in-between samples. • We evaluate our LH-Mix on three standard HTC datasets, demonstrating its effectiveness through the empirical performance. 2. Related Work 2.1. Hierarchical Text Classification (HTC) HTC refers to a specific type of multi-label text classification problem, where an instance lies in one or more paths from a taxonomic hierarchy in a top-down manner. The hierarchical structure is typically represented as a tree or a directed acyclic graph, where the root represents the highest-level label, and the leaf nodes correspond to the most specific labels. The key challenge in HTC lies in how to effectively utilize the large-scale, imbalanced, and structured label hierarchy. To perform HTC, various techniques can be employed. Previous works can be categorized into two types ( Zhou et al., 2020 ; Deng