Retrieval-augmented Encoders for Extreme Multi-label Text Classification
Yau-Shian Wang, Wei-Cheng Chang, Jyun-Yu Jiang, Jiong Zhang, Hsiang-Fu Yu, S. V. N. Vishwanathan · 2025
Read Retrieval-augmented Encoders for Extreme Multi-label Text Classification by Yau-Shian Wang, Wei-Cheng Chang, Jyun-Yu Jiang, Jiong Zhang, Hsiang-Fu Yu, S. V. N. Vishwanathan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10615.txt.
Introduction
Yau-Shian Wang and colleagues present retrieval-augmented encoders for extreme multi-label text classification. Their 2025 arXiv paper tackles tagging text from a huge label set. One-versus-all methods memorize head labels well through learned embeddings. Dual encoders generalize better to rare labels with little data. Past systems combined both, at the cost of complex training.
The new framework gives a dual encoder a retrieval ability. Retrieval supplies the memorization that dual encoders usually lack. No extra training pipeline is needed for that boost. Input text and label text still meet in a shared space. Rare labels benefit from the encoder's generalization strength. Common labels benefit from retrieved training signals. The authors aim for one simpler model with both virtues.
Catalog record
| CREATOR | Yau-Shian Wang, Wei-Cheng Chang, Jyun-Yu Jiang, Jiong Zhang, Hsiang-Fu Yu, S. V. N. Vishwanathan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-10615
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Retrieval-augmented Encoders for Extreme Multi-label Text Classification Yau-Shian Wang, Wei-Cheng Chang, Jyun-Yu Jiang, Jiong Zhang, Hsiang-Fu Yu, S. V. N. Vishwanathan Abstract Extreme multi-label classification (XMC) seeks to find relevant labels from an extremely large label collection for a given text input. To tackle such a vast label space, current state-of-the-art methods fall into two categories. The one-versus-all (OVA) method uses learnable label embeddings for each label, excelling at memorization (i.e., capturing detailed training signals for accurate head label prediction). In contrast, the dual-encoder (DE) model maps input and label text into a shared embedding space for better generalization (i.e., the capability of predicting tail labels with limited training data), but may fall short at memorization. To achieve generalization and memorization, existing XMC methods often combine DE and OVA models, which involves complex training pipelines. Inspired by the success of retrieval-augmented language models, we propose the Retrieval-augmented Encoders for XMC (RAE-XMC), a novel framework that equips a DE model with retrieval-augmented capability for efficient memorization without additional trainable parameter. During training, RAE-XMC is optimized by the contrastive loss over a knowledge memory that consists of both input instances and labels. During inference, given a test input, RAE-XMC retrieves the top- K K keys from the knowledge memory, and aggregates the corresponding values as the prediction scores. We showcase the effectiveness and efficiency of RAE-XMC on four public LF-XMC benchmarks. RAE-XMC not only advances the state-of-the-art (SOTA) DE method DEXML Gupta et al. (2024) , but also achieves more than 10x speedup on the largest LF-AmazonTitles-1.3M dataset under the same 8 A100 GPUs training environments. 1 Introduction Many real-world applications, such as e-commerce dynamic search advertising ( Prabhu et al., 2018 ) and semantic matching in product search ( Chang et al., 2021 ) , can be formulated as eXtreme Multi-label Classification (XMC) problems. These tasks involve retrieving relevant labels from a label collection of extremely large size L , typically under a tight latency budget. In such applications, L can be millions or more, and the label space is often long-tailed, posing a significant challenge in designing XMC methods that are good at labels with varying frequencies ( Dahiya et al., 2023a ; Gupta et al., 2024 ) . For head labels, it necessitates methods capable of memorization , which involves capturing all the detailed knowledge needed for accurate prediction. For tail labels, it requires generalization , as the model needs to capture general label representations that can be used to predict labels with few training instances. Traditional one-versus-all (OVA) methods prioritize memorization via trainable label classifiers, yet struggle to generalize effectively across tail labels ( Gupta et al., 2024 ) . Recently, the emergence of pre-trained language models (LMs) has demonstrated strong generalization power, prompting the adoption of dual-encoder (DE) architectures to effectively leverage label text for zero-shot XMC tasks ( Gupta et al., 2021 ; Xiong et al., 2022 ; Aggarwal et al., 2023 ) . However, without extensive training, these models may not excel at memorization, suggesting a pure DE model is not sufficient to achieve good performance at head labels of XMC tasks. To achieve both memorization and generalization power, competitive XMC approaches, such as \text{NGAME}_{\text{DE+OVA}} ( Dahiya et al., 2023a ) \text{DEXA}_{\text{DE+OVA}} ( Dahiya et al., 2023b ) , and OAK ( Mohan et al., 2024 ) usually follow a complex multi-stage solution. They first train a DE model that learns semantic embeddings between input and label text, which can generalize to unseen input queries and tail labels with text features. Then the DE model is further augmented by trainable one-versus-all (OVA) label classifiers, which enhance the memorization capability of head labels with diverse intent of queries. Recently, the pioneering work, DEXML ( Gupta et al., 2024 ) , achieved impressive results on several XMC benchmark datasets via a pure DE model, which significantly reduces the amount of trainable model parameters. While being a strong predictive model, DEXML requires extreme long training time to memorize the training signals of head labels, making it challenging to be applied to industrial applications. In this paper, we introduce Retrieval-augmented Encoders for XMC (RAE-XMC), a novel framework that equips a dual-encoder (DE) model with the retrieval-augmented capability. RAE-XMC compensates for the poor memorization capabilities of DE and eliminates the need for precise matching between input instance and label embedding spaces, thereby significantly reducing the training difficulty of XMC. Specifically, RAE-XMC trains the DE model to contrast over a joint knowledge memory space that consists of label descriptions and input instances. During the inference, given a test instance, RAE-XMC first retrieves the top b keys from the knowledge memory. RAE-XMC then generates predictions by aggregating the values (i.e., labels) of these keys based on their scores. The contributions of this paper are threefold as follows: • We introduce RAE-XMC, a novel retrieval-augmented framework for XMC problems, which enhances the underlying DE model with better memorization capability. Specifically, RAE-XMC offers controllable trade-off between memorization and generalization, namely the performance of head and tail label segments, respectively (cf., Section 5.3 ). • RAE-XMC significantly reduces the training difficulty for XMC problems in two ways. Firstly, it eliminates the need for training OVA label classifiers to memorize intricate complex patterns in head labels. Secondly, it alleviates meticulously training of the DE model to match label and input instance spaces (cf., Section 5.6 ). • We conducted extensive experiments to demonstrate the effectiveness and efficiency of RAE-XMC. Our approach not only advances the results of the current SOTA DE method, DEXML ( Gupta et al., 2024 ) , but also achieves performance on par with the SOTA XMC method, OAK ( Mohan et al., 2024 ) , without relying on external knowledge. Additionally, it offers significantly lower training time, achieving more than a 10x speedup on the largest LF-AmazonTitles-1.3M dataset (cf. Section 5.2 ). 2 Background Material We study the eXtreme Multi-label Text Classification (XMC) problem with label text. Consider a training set of N examples {\mathcal{D}}=\{(x_{i},{\mathbf{y}}_{i})\}_{i=1}^{N} , where x_{i}\in{\mathcal{X}} is the i -th input instance and {\mathbf{y}}_{i}\in\{0,1\}^{L} is the one-hot label vector with y_{i,\ell}=1 indicating that the label \ell is relevant to the input instance x_{i} . Depending on the availability of label text, \ell denotes either a label index or label text, respectively. The label space {\mathcal{L}} consists of |{\mathcal{L}}|=L labels. The goal of XMC is to learn a scoring function s_{\theta}:{\mathcal{X}}\times{\mathcal{L}}\rightarrow{\mathbb{R}} , parameterized by model parameters \theta , such that s_{\theta}(x,\ell) indicates the relevance between input instance x and label \ell . Model Parametrization. Based on the availability of label text features, there are two model families for the XMC task: the One-versus-all (OVA) classifier and the Dual-Encoder (DE) model. The OVA classifier, which ignores the label text features, consists of an input text encoder f_{\phi}:{\mathcal{X}}\rightarrow{\mathbb{R}}^{d} and a classification matrix {\mathbf{W}}\in{\mathbb{R}}^{L\times d} . The scoring function of the OVA classifier then becomes s_{\theta}(x,\ell)=\langle{\mathbf{x}},{\mathbf{w}}_{\ell}\rangle where {\mathbf{x}}=f_{\phi}(x) is the input query embedding and {\mathbf{w}}_{\ell} (the \ell th row of {\mathbf{W}} ) is the classification vector for label \ell . For the OVA classifier, the trainable parameter \theta=[\phi;{\mathbf{W}}] scales linearly to the size of label space L , which can be large and difficult to train. In contrast, our paper studies the DE model,