R1-RE: Cross-Domain Relationship Extraction with RLVR

Runpeng Dai 1 Tong Zheng 2 Run Yang 3 Hongtu Zhu 1† 1 University of North Carolina at Chapel Hill 2 University of Maryland, College Park 3 BiliBili {runpeng, htzhu}@email.unc.edu [email protected] [email protected] · 2025

Read R1-RE: Cross-Domain Relationship Extraction with RLVR by Runpeng Dai 1 Tong Zheng 2 Run Yang 3 Hongtu Zhu 1† 1 University of North Carolina at Chapel Hill 2 University of Maryland, College Park 3 BiliBili {runpeng, htzhu}@email.unc.edu [email protected] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04642.txt.

Introduction

Runpeng Dai and co-authors teach small language models to pull links between named things from text, in a 2025 study. Older systems map text straight to labels and often fail on new topics. The new method, R1-RE, instead trains the model to reason like a human coder who follows written guidelines.

Training uses rewards that can be checked as right or wrong, which builds step-by-step habits. On a public test set and a private medical one, the 7B model hit about 70 percent accuracy on unseen topics. That score matches large closed models such as GPT-4o. The authors also track how the reasoning habits form during training, which helps others copy the approach.

Catalog record

CREATORRunpeng Dai 1 Tong Zheng 2 Run Yang 3 Hongtu Zhu 1† 1 University of North Carolina at Chapel Hill 2 University of Maryland, College Park 3 BiliBili {runpeng, htzhu}@email.unc.edu [email protected] [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04642

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

R1-RE: Cross-Domain Relationship Extraction with RLVR Runpeng Dai 1 Tong Zheng 2 Run Yang 3 Hongtu Zhu 1† 1 University of North Carolina at Chapel Hill 2 University of Maryland, College Park 3 BiliBili {runpeng, htzhu}@email.unc.edu [email protected] [email protected] Abstract Relationship extraction (RE) is a core task in natural language processing. Traditional approaches typically frame RE as a supervised learning problem, directly mapping context to labels—an approach that often suffers from poor out-of-domain (OOD) generalization. Inspired by the workflow of human annotators, we reframe RE as a reasoning task guided by annotation guidelines and introduce R1-RE , the first reinforcement learning with verifiable reward (RLVR) framework for RE tasks. Our method elicits the reasoning abilities of small language models for annotation tasks, resulting in significantly improved OOD robustness. We evaluate our approach on the public Sem-2010 dataset and a private MDKG dataset. The R1-RE-7B model attains an average OOD accuracy of approximately 70%, on par with leading proprietary models such as GPT-4o. Additionally, our comprehensive analysis provides novel insights into the training dynamics and emergent reasoning behaviors of the RLVR paradigm for RE. 1 Introduction Relationship extraction (RE) ( Zhao et al. 2024 ) is a fundamental task in natural language processing (NLP) that involves either classifying the relationships between pairs of entities (relationship classification) or extracting (subject, relation, object) triples (triplet extraction) from context. RE serves as a foundation for numerous downstream applications ( Nayak et al. 2021 ) , most notably in knowledge graph (KG) construction ( Zhong et al. 2023 ) . KGs have demonstrated broad utility across domains such as biomedical research ( Yang et al. 2024 ) and e-commerce ( Li et al. 2020 ) , and are increasingly recognized for their potential to enhance LLMs—for example, through approaches like GraphRAG ( Edge et al. 2024 ) and graph-based reasoning ( Luo et al. 2023 ) . Consequently, developing robust and effective RE methods is of growing importance. Figure 1: Testing accuracy on the MDKG dataset for R1-RE-7B trained on the Sem-2010 dataset, compared with other models. Detailed results are provided in Table 3 . Standard approaches to RE typically adopt a pretraining-and-finetuning paradigm, employing mid-sized pre-trained models such as BART ( Lewis et al. 2019 ) and BERT ( Devlin et al. 2019 ) . While these models are lightweight, they require substantial fine-tuning to generalize to out-of-domain scenarios. Recently, there has been growing interest in leveraging LLMs for RE, capitalizing on their rich language understanding ( Zeng et al. 2023 ) and impressive in-context learning capabilities ( Brown et al. 2020 ) . In this context, some studies have explored few-shot prompting strategies with API models 1 1 1 API models refer to proprietary, closed-source models that are typically accessible only via external APIs. ( Wadhwa et al. 2023 ; Xu et al. 2023 ) , while others have investigated supervised fine-tuning (SFT) methods ( Ettaleb et al. 2025 ) using lighter-weight, open-source LLMs for RE tasks. As illustrated in Figure 2 (a), few-shot learning yields only marginal improvements over the original model. Furthermore, Figure 2 (b) demonstrates that the naive SFT approach can boost in-domain performance but provides limited generalization to out-of-domain scenarios, suggesting that SFT primarily promotes memorization rather than genuine annotation ability ( Chu et al. 2025 ) . This observation raises a critical question: How to enhance the RE capability of small LLMs for robust cross-domain performance? Figure 2: (a) Test accuracy on two RE datasets with varying numbers of few-shot examples. (b) In-domain and out-of-domain test accuracy of the SFT model using a naive prompt (Figure 3 ) versus a prompt incorporating the annotation guide (Figure 4 ). To this end, we examined the workflow of human annotators (see Figure 3 ). Annotators are typically provided with detailed annotation guidelines and iteratively compare the target sentence against these guidelines—formulating hypotheses, verifying them, and ultimately reaching a conclusion. While entities and relationship types may vary across domains, the reasoning skills developed through annotation guideline usage are broadly generalizable. Motivated by this intuition, we reconceptualize RE as a reasoning task grounded in annotation guidelines. To this end, we modify the standard SFT paradigm by explicitly incorporating the annotation guide into the prompt. As shown in Figure 2 , this results in an approximately 10% improvement in out-of-domain RE performance. However, a substantial gap remains when compared to proprietary API models. To this end, we propose R1-RE to enhance the reasoning abilities of LLMs for RE tasks. Our framework is inspired by the recent success of reinforcement learning with verifiable reward (RLVR), which has proven effective in promoting reasoning skills in smaller models on challenging domains such as mathematics and code generation ( Guo et al. 2025 ) . We evaluate our proposed method on both a public dataset (SemEval-2010 Task 8) and a private dataset (MDKG). Our results show that R1-RE improves the OOD performance of Qwen-7B-Instruct by approximately 30%, achieving results comparable to those of leading proprietary models. Further analysis demonstrates that incorporating additional training data can further enhance performance. 2 Preliminaries 2.1 Task Definition Let \mathcal{S} denote the input sentence, and let \mathcal{E} and \mathcal{R} represent the predefined sets of entity types and relationship types, respectively. Let \mathcal{K} denote the annotation guideline for the task, which provides the definitions of \mathcal{E} and \mathcal{R} as well as other annotation instructions. We consider two relationship extraction (RE) tasks: Relation Classification (RC) and Triplet Extraction (TE). In this work, we primarily focus on RC tasks; the definition of the TE task is provided in Appendix B . Relationship Classification(RC) : In this task, the subject-object pair ( e_{\text{sub}} , e_{\text{obj}} ) has already been identified from \mathcal{S} . The goal is to assign a relationship type y\in\mathcal{Y} , where \mathcal{Y} is the predefined set of relationship types, based on the sentence \mathcal{S} and the annotation guideline \mathcal{K} . Each sentence \mathcal{S} contains exactly one entity pair. For instance, Sentence <e1>Counseling interventions</e1> can be effective in preventing <e2>perinatal depression</e2>. Label treatment-for(e1, e2) 2.2 Group Relative Policy Optimization (GRPO) In this work, we formulate the language generation process of LLMs as a sequential Decision Process and optimize the LLM with the Group Relative Policy Optimization (GRPO) algorithm ( Shao et al. 2024 ) . Specifically, let \pi_{\theta} denote the LLM with parameters \theta . At each training step, given a prompt q sampled from the dataset \mathcal{D} , we use \pi_{\theta} to generate a group of G candidate outputs, denoted as {o_{1},o_{2},\ldots,o_{G}} . For each candidate output, we compute the corresponding reward r_{1},r_{2},\ldots,r_{G} by comparing the output with the gold standard. The advantage at the t -th token of the i -th output is then calculated as A_{i,t}=A_{i}=\frac{r_{i}-\operatorname{mean}(r_{1},r_{2},\ldots,r_{G})}{\operatorname{std}(r_{1},r_{2},\ldots,r_{G})}. Let \pi_{\theta_{\mathrm{old}}} denote the model from the previous training step, and \pi_{\mathrm{ref}} denote the original model prior to training. GRPO maximizes the following objective function to optimize \pi_{\theta} : \displaystyle\mathbb{E}_{q\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}}\biggl[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|} \displaystyle\min\Bigl(\frac{\pi^{i,t}_{\theta}}{\pi^{i,t}_{\theta_{\mathrm{old}}}}\,A_{i},\,\mathrm{clip}(\frac{\pi^{i,t}_{\theta}}{\pi^{i,t}_{\theta_{\mathrm{old}}}},1-\varepsilon,\,1+\varepsilon)A_{i}\Bigr) \displaystyle-\,\beta\,D_{\mathrm{KL}}\bigl(\pi_{\theta}\,\big\|\,\pi_{\mathrm{ref}}\bigr)\biggr], where \pi^{i,t}_{\theta}=\pi_{\theta}(o_{i,t}\mid q,o_{i,1}\ldots,o_{i,t-1}) , and similarly for \pi^{i,t}_{\theta{\mathrm{old}}} . The hyperparameters \varepsilon and \beta control the ratio clipping threshold and the weight of the Kullback–Leibler (KL) divergence penalty, respectively. Specifically, ratio clipping mitigates the risk of large, destabilizing policy updates, while the KL penalty constrains the updated policy from deviating excessively from the reference

Read the complete text