Few-Shot, No Problem: Descriptive Continual Relation Extraction Nguyen Xuan Thanh, Anh Duc Le, Quyen Tran, Thanh-Thien Le, Linh Ngo Van, Thien Huu Nguyen Abstract Few-shot Continual Relation Extraction is a crucial challenge for enabling AI systems to identify and adapt to evolving relationships in dynamic real-world domains. Traditional memory-based approaches often overfit to limited samples, failing to reinforce old knowledge, with the scarcity of data in few-shot scenarios further exacerbating these issues by hindering effective data augmentation in the latent space. In this paper, we propose a novel retrieval-based solution, starting with a large language model to generate descriptions for each relation. From these descriptions, we introduce a bi-encoder retrieval training paradigm to enrich both sample and class representation learning. Leveraging these enhanced representations, we design a retrieval-based prediction method where each sample ”retrieves” the best fitting relation via a reciprocal rank fusion score that integrates both relation description vectors and class prototypes. Extensive experiments on multiple datasets demonstrate that our method significantly advances the state-of-the-art by maintaining robust performance across sequential tasks, effectively addressing catastrophic forgetting. 1 Introduction Relation Extraction (RE) refers to classifying semantic relationships between entities within text into predefined types. Conventional RE tasks assume all relations are present at once, ignoring the fact that new relations continually emerge in the real world. Few-shot Continual Relation Extraction (FCRE) is a subfield of continual learning ( Hai et al. 2024 ; Van et al. 2022 ; Phan et al. 2022 ; Tran et al. 2024a ; Tran et al. 2024b ; Le et al. 2024a ) where a model must continually assimilate new emerging relations while avoiding the forgetting of old ones, a task made even more challenging by the limited training data available. The importance of FCRE stems from its relevance to dynamic real-world applications, garnering increasing interest in the field ( Chen, Wu, and Shi 2023a ; Le et al. 2024c ; Le et al. 2025 ) . Figure 1: Existing FCRE methods face catastrophic forgetting due to the limited and poor quality of old training samples stored in the memory buffer. State-of-the-art approaches to FCRE often rely on memory-based methods for continual learning ( Lopez-Paz and Ranzato 2017 ; Nguyen et al. 2023 ; Le et al. 2024b ; Dao et al. 2024 ) . However, these methods frequently suffer from overfitting to the limited samples stored in memory buffers. This overfitting hampers the reinforcement of previously learned knowledge, leading to catastrophic forgetting—a marked decline in performance on learnt relations when new ones are introduced (Figure 1 ). The few-shot scenario of FCRE exacerbates these issues, as the scarcity of data not only impedes learning on new tasks, but also hinders helpful data augmentation, which are crucial in many methods ( Shin et al. 2017 ) . In order to improve on these methods, we must not completely disregard them or dwell on their weaknesses, but rather contemplate their biggest strength. Why do so many methods use the memory buffer in the first place? The primary objective of these replay buffers is to rehearse and reinforce past knowledge, providing the model with something to ”look back” at during training. However, these past samples may not always be representative of the entire class and can still lead to sub-optimal performance. Based on this observation, we propose a straightforward: besides relying on potentially unrepresentative past samples, we leverage our knowledge of the past relations themselves. This insight leads to our approach of generating detailed descriptions for each relation. These descriptions inherently represent the class more accurately than the underlying information from a set of samples, serving as stable pivots for the model to align with past knowledge while learning new information. By using these descriptions, we create a more robust and effective method for Few-Shot Continual Relation Extraction, ensuring better retention of knowledge across tasks. Overall, our paper makes the following contributions: a. We introduce an innovative approach to Few-Shot Continual Relation Extraction that leverages Large Language Models (LLMs) to generate comprehensive descriptions for each relation. These descriptions serve as stable class representations in the latent space during training. Unlike the variability and limitations of a limited set of samples from the memory buffer, these descriptions define the inherent meaning of the relations, offer a more reliable anchor, significantly reducing the risk of catastrophic forgetting. Importantly, LLMs are employed exclusively for generating descriptions and do not participate in the training or inference processes, ensuring that our method incurs minimal computational overhead. b. We design a bi-encoder retrieval learning framework for both sample and class representation learning. In addition to sample representation contrastive learning, we integrate a description-pivot learning process, ensuring alignment of samples which maximize their respective class samples proximity, while non-matching samples are distanced. c. Building on the enhanced representations, we introduce the Descriptive Retrieval Inference (DRI) strategy. In this approach, each sample ”retrieves” the most fitting relation using a reciprocal rank fusion score that integrates both class descriptions and class prototypes, effectively finalizing the retrieval-based paradigm that underpins our method. 2 Background 2.1 Problem Formulation In Few-Shot Continual Relation Extraction (FCRE), a model must continuously assimilate new knowledge from a sequential series of tasks. For each t -th task, the model undergoes training on the dataset D^{t}=\{(x^{t}_{i},y^{t}_{i})\}_{i=1}^{N\times K} . Here, N represents the number of relations in the task R^{t} , and K denotes the limited number of samples per relation, reflecting the few-shot learning scenario. Each sample (x,y) includes a sentence x containing a pair of entities (e_{h},e_{t}) and a relation label y\in R . This type of task setup is referred to as ”N-way-K-shot” ( Chen, Wu, and Shi 2023a ) . Upon completion of task t , the dataset D^{t} should not be extensively included in subsequent learning, as continual learning aims to avoid retraining on all prior data. Ultimately, the model’s performance is assessed on a test set which encompasses all encountered relations \tilde{R}^{T}=\bigcup_{t=1}^{T}R^{t} . For clarity, each task in FCRE can be viewed as a conventional relation extraction problem, with the key challenge being the scarcity of samples available for learning. The primary goal of FCRE is to develop a model that can consistently acquire new knowledge from limited data while retaining competence in previously learned tasks. In the following subsections, we will explore the key aspects of FCRE models as addressed by state-of-the-art studies. 2.2 Encoding Latent Representation A key initial consideration in Relation Extraction is how to formalize the latent representation of the input, as the output of a Transformer ( Vaswani et al. 2017 ) is a matrix. In this work, we adopt a method recently introduced by Ma et al. 2024 . Given an input sentence x , which includes a head entity e_{h} and a tail entity e_{t} , we reformulate it into a Cloze-style phrase T(x) by incorporating a [MASK] token, which represents the relation between the entities. Specifically, the template is structured as follows: \displaystyle\begin{aligned} T({x})=\;&x\left[v_{0:n_{0}-1}\right]e_{h}\left[v_{n_{0}:n_{1}-1}\right][\texttt{MASK}]\\ &\left[v_{n_{1}:n_{2}-1}\right]e_{t}\left[v_{n_{2}:n_{3}-1}\right].\end{aligned} (1) Each [v_{i}] denotes a learnable continuous token, and n_{j} determines the number of tokens in each phrase. In our specific implementation, we use BERT’s [UNUSED] tokens as [v] . The soft prompt phrase length is set to 3 tokens, meaning n_{0} , n_{1} , n_{2} and n_{3} correspond to the values of 3 , 6 , 9 , and 12 , respectively. We then forward the templated sentence T({x}) through BERT to encode it into a sequence of continuous vectors, from which we obtain the hidden representation \bm{z} of the input, corresponding to the position of the [MASK] token: \bm{z}=[\mathcal{M}\circ T]({x})[\rm{position}(\texttt{[MASK]})], (2) where \mathcal{M} denotes the backbone pre-trained language model. This latent representation is then passed through an MLP for prediction, enabling the model to learn which relation that best fills the [MASK] token. 2.3 Learning Latent Representation In conventional Relation Extraction scenarios, a basic framework typically employs a backbone PLM followed by an MLP classifier to directly map the input space to the label space using Cross Entropy Loss. However, this approach proves inadequate in data-scarce settings ( Snell, Swersky, and Zemel 2017 ) . Consequently, training paradigms which directly target the latent space, such as contrastive learning, emerge as more suitable approaches. To enhance the semantic richness of the information extracted from the training samples, two popular losses are often utilized: Supervised Contrastive Loss and Hard Soft Margin Triplet Loss . Supervised Contrastive Loss. To enhance the model’s discriminative capability, we employ the Supervised Contrastive Loss (SCL) ( Khosla et al. 2020 ) . This loss function is designed to bring positive pairs of samples, which share the same class label, closer together in the latent space. Simultaneously, it pushes negative pairs, belonging to different classes, further apart. Let \bm{z}_{x} represent the hidden vector output of sample {x} , the positive pairs (\bm{z}_{x},\bm{z}_{p}) are those who share a class, while the negative pairs (\bm{z}_{x},\bm{z}_{n}) correspond to different labels. The SCL is computed as follows: \displaystyle\mathcal{L}_{\textrm{SC}}(x)=-\sum_{{p}\in P({x})}\log\frac{f(\bm{z}_{x},\bm{z}_{p})}{\sum_{{u}\in\mathcal{D}\setminus\{x\}}f(\bm{z}_{x},\bm{z}_{u})} (3) where f(\mathbf{x},\mathbf{y})\coloneqq\exp\left(\frac{\gamma(\mathbf{x},\mathbf{y})}{\tau}\right) , \gamma(\cdot,\cdot) denotes the cosine similarity function, and \tau is the temperature scaling hyperparameter. P({x}) and \mathcal{D} denote the sets of positive samples with respect to sample {x} and the training set, respectively. Hard Soft Margin Triplet Loss. To achieve a balance between flexibility and discrimination, the Hard Soft Margin Triplet Loss (HSMT) integrates both hard and soft margin triplet loss concepts ( Hermans, Beyer, and Leibe 2017 ) . This loss function is designed to maximize the separation between the most challenging positive and negative samples, while preserving a soft margin for improved flexibility. Formally, the loss is defined as: \mathcal{L}_{\textrm{ST}}({x})=\\ -\log\bigg(1\ +\max_{{p}\in P(\bm{x})}e^{\xi(\bm{z}_{x},\bm{z}_{p})}-\min_{{n}\in N({x})}e^{\xi(\bm{z}_{x},\bm{z}_{n})}\bigg), (4) where \xi(\cdot,\cdot) denotes the Euclidean distance function. The objective of this loss is to ensure that the hardest positive sample is as distant as possible from the hardest negative sample, thereby enforcing a flexible yet effective margin. During training, these two losses is aggregated and referred to as the Sample-based learning loss : \mathcal{L}_{\text{Samp}}=\beta_{\text{SC}}\cdot\mathcal{L}_{\text{SC}}+\beta_{\text{ST}}\cdot\mathcal{L}_{\text{ST}} (5) 3 Proposed Method 3.1 Label Descriptions Figure 2: Prompt to generate relation descriptions with LLMs. A core component of our method is achieving robust class latent representations, making class encoding crucial. To this end, having detailed definitions for each label, alongisde the hidden information extracted from the samples, is essential for our approach. In fact, the datasets used for benchmarking already provide each relation with a concise description, which we refer to as the Raw description . While leveraging these descriptions has shown promise in previous work ( Luo et al. 2024 ) , this approach remains limited due to its reliance on a one-to-one mapping between input embeddings and a single label description representation per task. This singular approach fails to offer rich, diverse, and robust information about the labels, leading to potential noise, instability, and suboptimal model performance. To address these limitations, we employ Gemini 1.5 ( Team et al. 2023 ; Reid et al. 2024 ) to generate K diverse, detailed, and illustrative descriptions for each relation. In particular, for each label, the respective raw description will be fed into the LLM prompt , serving as an expert-in-the-loop to guide the model. Our prompt template is depicted in Figure 2 . 3.2 Description-pivot Learning The single most valuable quality of class descriptions in our problem is that they are literal definitions of a relation, which makes them more accurate representations of that class than the underlying information from a set of samples. Thanks to this strength, they serve as stable knowledge anchors for the model to rehearse from, enabling effective reinforcement of old knowledge while assimilating new information. Unlike the variability of individual samples, a description remains consistent, providing a more reliable reference point for the model to rehearse from, effectively mitigating catastrophic forgetting. To fully leverage this inherent advantage, we integrate these descriptions into the training process, framing the task as one of retrieving definition, which embodies real-world meaning, rather than a straightforward categorical classification. By doing so, we capitalize on the unchanging nature of descriptions, making them the focal point of our model’s learning. Specifically, we incorporate two description-centric losses to enhance this retrieval-oriented approach: \mathcal{L}_{\text{Des}}=\beta_{\text{HM}}\cdot\mathcal{L}_{\text{HM}}+\beta_{\text{MI}}\cdot\mathcal{L}_{\text{MI}}. (6) Here, \mathcal{L}_{\text{HM}} and \mathcal{L}_{\text{MI}} denote the Hard Margin Loss and the Mutual Information Loss, respectively. These losses are elaborated upon in the following paragraphs. Hard Margin Loss. The Hard Margin Loss leverages label descriptions to refine the model’s ability to distinguish between hard positive and hard negative pairs. Given the output hidden vectors \{\bm{d}^{k}_{x}\}_{k=1,...,K} from BERT corresponding to the label description of sample {x} , and \bm{z}_{p} and \bm{z}_{n} representing the hidden vectors of positive and negative samples respectively, the loss function is formulated to maximize the alignment between \bm{d}^{k}_{x} and its corresponding positive sample, while enforcing a strict margin against negative samples. Specifically, the loss is formulated as follows: \displaystyle\mathcal{L}_{\textrm{HM}}(x) \displaystyle=\sum_{k=1}^{K}\mathcal{L}_{\textrm{HM}}^{k}(x), (7) \displaystyle\mathcal{L}_{\textrm{HM}}^{k}(x) \displaystyle=\sum_{p\in P_{\textrm{H}}(x)}(1-\gamma(\bm{d}^{k}_{x},\bm{z}_{p}))^{2} \displaystyle+\sum_{n\in N_{\textrm{H}}(x)}max(0,m-1+\gamma(\bm{d}^{k}_{x},\bm{z}_{n}))^{2}, (8) where m is a margin hyperparameter; \gamma(\cdot,\cdot) denotes the cosine similarity function; P_{\textrm{H}}(x) and N_{\textrm{H}}(x) represent the sets of hard positive and hard negative samples, respectively. They are determined by comparing the similarity between \bm{d}^{k}_{x} and both positive and negative pairs, specifically focusing on the most challenging pairs where the similarity to negative samples is close to or greater than that of positive samples, defined as follows: \displaystyle\begin{aligned} P_{\textrm{H}}(x)=\{&p\in P(x)|1-\gamma(\bm{d}^{k}_{x},\bm{z}_{p})\\ &>min_{n\in N(x)}(1-\gamma(\bm{d}^{k}_{x},\bm{z}_{n})),\forall k\in[K]\},\end{aligned} (9) \displaystyle\begin{aligned} N_{\textrm{H}}(x)=\{&n\in N(x)|1-\gamma(\bm{d}^{k}_{x},\bm{z}_{n})\\ &