Improving Generalization in Intent Detection: GRPO with Reward-Based Curriculum Sampling

Zihao Feng, Xiaoxue Wang, Ziwei Bai, Donghang Su, Bowen Wu, Qun Yu, Baoxun Wang · 2025

Read Improving Generalization in Intent Detection: GRPO with Reward-Based Curriculum Sampling by Zihao Feng, Xiaoxue Wang, Ziwei Bai, Donghang Su, Bowen Wu, Qun Yu, Baoxun Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13592.txt.

Introduction

Chatbots that book and schedule must guess what the user wants, and they stumble on requests they have never seen. Zihao Feng and six co-authors attack that stumble with reinforcement learning in a 2025 paper. Their twist is a sampling scheme that steers training toward the cases the model currently gets wrong.

Models trained this way generalized to unseen intents far better than the same models fine-tuned on examples alone. The reward-based sampling added a further gain by spending training effort where errors clustered. Writing out a chain of thought before answering helped most on the hardest, most tangled requests. The authors frame the recipe as practical guidance for dialogue systems that must absorb new tools and tasks without retraining from scratch.

Catalog record

CREATORZihao Feng, Xiaoxue Wang, Ziwei Bai, Donghang Su, Bowen Wu, Qun Yu, Baoxun Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13592

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Generalization in Intent Detection: GRPO with Reward-Based Curriculum Sampling Zihao Feng, Xiaoxue Wang, Ziwei Bai, Donghang Su, Bowen Wu, Qun Yu, Baoxun Wang Abstract Intent detection, a critical component in task-oriented dialogue (TOD) systems, faces significant challenges in adapting to the rapid influx of integrable tools with complex interrelationships. Existing approaches, such as zero-shot reformulations and LLM-based dynamic recognition, struggle with performance degradation when encountering unseen intents, leading to erroneous task routing. To enhance the model’s generalization performance on unseen tasks, we employ Reinforcement Learning (RL) combined with a Reward-based Curriculum Sampling (RCS) during Group Relative Policy Optimization (GRPO) training in intent detection tasks. Experiments demonstrate that RL-trained models substantially outperform supervised fine-tuning (SFT) baselines in generalization. Besides, the introduction of the RCS, significantly bolsters the effectiveness of RL in intent detection by focusing the model on challenging cases during training. Moreover, incorporating Chain-of-Thought (COT) processes in RL notably improves generalization in complex intent detection tasks, underscoring the importance of thought in challenging scenarios. This work advances the generalization of intent detection tasks, offering practical insights for deploying adaptable dialogue systems. Figure 1 : Comparative performance of RL-Trained and SFT-Trained models in intent detection across various generalization scenarios 1 Introduction As a crucial component of Task-oriented Dialogue (TOD) systems gupta2024dard ; xu2024rethinking , the intent detection module aims to identify the underlying requirements of users’ queries weld2022survey ; casanueva2020efficient . Consequently, the intent detection models are expected to efficiently adapt to evolving task priorities and requirements, so as to conduct dynamic task allocation among multiple agents in complicated application scenarios. The recent development of LLMs has accelerated the evolution of TOD systems, and with the rapid iteration of integrable artificial API tools du2024anytool ; qu2024chatgpt , the number of AI tools that can be incorporated into TOD systems is increasing. This situation leads to a great challenge that, actually, intent detection models need to flexibly adapt to newly introduced tools for unseen tasks, with no timely incremental training processes. In many cases, tools within the management of intent detection modules maintain complex interrelationships, such as functional similarity, overlapping, inclusion, etc. Thus, the generalization of intent detection models is the essence for TOD systems to adjust to complicated practical scenarios, in which a number of tools, with complex relationships and interactions, may be frequently involved. Previous studies have made much efforts to improve the accuracy of intent detection models by adopting new tools to handle unseen tasks. For example, the model proposed by Siddique et al. introduces external common sense knowledge to address this problem DBLP:conf/sigir/SiddiqueJXH21 . Comi et al. DBLP:conf/emnlp/ComiCPM23 reformatted the tasks in an NLI format to achieve zero-shot capability. Moreover, LLM-based models parikh2023exploring ; gupta2024dard dynamically recognized unknown tasks by capitalizing on their inherent zero-shot capabilities. However, these models often experienced significant performance degradation in intent detection models when confronted with unseen or new intent, resulting in the system incorrectly routing user intent to the unmatched agent. This situation indicates that enhancing the generalization of intent detection models is particularly critical. Reinforcement learning has been proved to be valuable in improving the generalization of LLMs swamy2025roadsleadlikelihoodvalue , which has also been supported by the exceptional cross-task generalization of the recent model DeepSeek-R1 guo2025deepseek . Inspired by the principle of DeepSeek-R1, we propose to apply the Group Relative Policy Optimization (GRPO) methodology to enhance the generalization of the intent detection model. In particular, to ensure that the R1-style RL process achieves expected performances on the intent detection problem, a sampling strategy is presented in this work. As depicted in Figure 1 , the experimental results demonstrate that in varying generalization scenarios, the reinforcement learning (RL) model successfully predicts user query intents, significantly outperforming the supervised fine-tuned (SFT) model. This superiority is particularly evident in terms of generalization across unseen intents, subdivided intents, grouped intents, and cross-language. In conclusion, our work offers the following findings: • We demonstrate that models trained with RL significantly outperform those trained with SFT on the intent detection problem, in terms of generalization across unseen intents, subdivided intents, grouped intents, and cross-language. • To stimulate the capability of GRPO training, we introduce the Rewards-based Curriculum Sampling Strategy, which is found to be valuable for enabling models to focus more on challenging cases during the training process. • Incorporating COT wei2022chain processes during reinforcement learning significantly enhances model generalization on complex intent detection tasks, highlighting the importance of thought processes for improving generalization in challenging scenarios. • Furthermore, our experiments also show that even a base model without instruction training can achieve performance comparable to the instruction model on the intent detection task. This finding suggests that the Function Call capability of the base model may not be a necessary prerequisite for intent detection models trained with RL. 2 Method 2.1 Task Formulation In task-oriented dialogue systems, accurate detection of user intents is essential for dialogue state tracking and subsequent API execution. We formulate the intent detection task as follows: Given a dialogue history H=\{(u_{1},a_{1},y_{1}),(u_{2},a_{2},y_{2}),\ldots,(u_{t-1},a_{t-1},y_{t-1})\} , where u_{i} , a_{i} , and y_{i}\in\mathcal{Y} represent the user’s utterance, the assistant’s response, and the ground truth intent label at turn i , respectively. \mathcal{Y}=\{c_{1},c_{2},\ldots,c_{K}\} denotes a predefined set of K actionable intents related to domain-specific operations, with each intent c_{i} associated with a natural language description d_{i} in the prompt. The objective of an intent detection model M is to accurately predict the intent y_{t}\in\mathcal{Y} of the current user’s utterance u_{t} . Formulated as: \mathcal{L}(\theta)=-\frac{1}{N}\sum_{n=1}^{N}\log P_{\theta}(y_{t}^{n}|H^{n},u_{t}^{n},{d_{1},d_{2},\ldots,d_{K}}) (1) where \theta represents the parameters of model M , N is the number of training examples, P_{\theta} denotes the probability assigned by model M . Apparently, the model M demonstrates non-trivial generalization potential for evolving dialogue systems, as its architecture theoretically supports the discovery of novel intent categories through dynamic prompt engineering. Formally, for y_{t}=c_{K+1}\notin\mathcal{Y} , model M can add the description d_{K+1} of c_{K+1} to the prompt to predict y_{t} . In particular, this y_{t} may represent not only a completely new category distinct from \mathcal{Y} , but also potentially a division or recombination of previous categories. 2.2 Intent Detection via Reinforcement Learning Directly applying supervised fine-tuning (SFT) to learn the prediction of y_{t} has been a conventional approach, however, this method often suffers from poor generalization capabilities. In this paper, inspired by DeepSeek-R1-Zero guo2025deepseek , which demonstrated the significant potential of reinforcement learning combined with model reasoning, we design rule-based rewards and exclusively employ GRPO shao2024deepseekmath to guide model training. Specifically, building upon an arbitrary LLM, we construct a complete prompt using the ReAct Prompting yao2023react method, where the system prompt is "You are a helpful assistant.". In the final turn of the dialogue, we insert an instruction composed of the user query and other relevant information, such as descriptions of available tools. The specific instruction template is as follows. Instruction Template of ReAct Prompting You are an agent that helps users choose the right tool or tools from the list of given tools to solve their problems. For each tool, you are first given its description and required parameters. Then, a logic module specifically explains the logical information needed for this tool to handle multi-turn conversation issues. ## Tool APIs {tools text} ## Task Logic {logic text}

Read the complete text