An Efficient Task-Oriented Dialogue Policy: Evolutionary Reinforcement Learning Injected by Elite Individuals

Yangyang Zhao, Ben Niu, Libo Qin, Shihan Wang · 2025

Read An Efficient Task-Oriented Dialogue Policy: Evolutionary Reinforcement Learning Injected by Elite Individuals by Yangyang Zhao, Ben Niu, Libo Qin, Shihan Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03519.txt.

Introduction

Yangyang Zhao, Ben Niu, Libo Qin, and Shihan Wang blend evolution search with deep reinforcement learning to train chat agents. Their 2025 paper targets task-based dialogue, where an agent must steer a chat to a goal. Deep learning agents often stall in poor local habits because the state and action space is huge.

Evolution methods keep a diverse pool of candidate agents, which aids broad search. The blend pairs that breadth with the fine local tuning of deep learning. Loose language in dialogue slows raw evolution, though. An elite injection step drops top agents into the pool at smart moments to speed the hunt. Tests on four data sets show clear gains in balance and final scores.

Catalog record

CREATORYangyang Zhao, Ben Niu, Libo Qin, Shihan Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03519

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Efficient Task-Oriented Dialogue Policy: Evolutionary Reinforcement Learning Injected by Elite Individuals Yangyang Zhao, Ben Niu, Libo Qin, Shihan Wang Abstract Deep Reinforcement Learning (DRL) is widely used in task-oriented dialogue systems to optimize dialogue policy, but it struggles to balance exploration and exploitation due to the high dimensionality of state and action spaces. This challenge often results in local optima or poor convergence. Evolutionary Algorithms (EAs) have been proven to effectively explore the solution space of neural networks by maintaining population diversity. Inspired by this, we innovatively combine the global search capabilities of EA with the local optimization of DRL to achieve a balance between exploration and exploitation. Nevertheless, the inherent flexibility of natural language in dialogue tasks complicates this direct integration, leading to prolonged evolutionary times. Thus, we further propose an elite individual injection mechanism to enhance EA’s search efficiency by adaptively introducing best-performing individuals into the population. Experiments across four datasets show that our approach significantly improves the balance between exploration and exploitation, boosting performance. Moreover, the effectiveness of the EII mechanism in reducing exploration time has been demonstrated, achieving an efficient integration of EA and DRL on task-oriented dialogue policy tasks. * * footnotetext: Corresponding author 1 Introduction Task-oriented dialogue (TOD) systems aim to understand user intent ( Qin et al. 2021 ) , generate responses ( Qin et al. 2023 ) , and steer conversations toward goals, with the dialogue policy (DP) ( Kwan et al. 2019 ) playing a crucial role in determining and steering conversations. Deep Reinforcement Learning (DRL) has proven effective in optimizing DPs due to its strengths in sequential decision-making ( Kwan et al. 2023 ) . While Large Language Models ( Chung et al. 2023 ; Hu et al. 2024 ; Kumar 2024 ) are increasingly used in TOD systems for their language capabilities, they are rarely applied in DPs because of their limited decision-making ability and the high resource demands for prompt design and fine-tuning ( Yao et al. 2024 ; Yang et al. 2024 ) . Thus, optimizing DPs through DRL remains the primary approach ( Du et al. 2024 ) . However, DRL struggles to balance exploration and exploitation. Insufficient exploration can lead to local optima, while excessive exploration results in inefficient training 1 1 1 We have shown that the above observation also applies to the dialogue task, with experimental results and analysis provided in Appendix C . ( Henderson et al. 2018 ) . Figure 1: Framework of the EIERL method, comprising two modules: exploration and exploitation. The exploration module aims to form an EA population (multiple individuals a) to explore diverse experiences, while the exploitation module aims to form a DRL population (multiple individuals b) to use existing experiences for training. Research on balancing exploration and exploitation in DRL-based DP can be categorized into two main approaches: direct and indirect enhancement of exploration efficiency. Direct methods focus on designing effective exploration strategies that, while powerful, are often limited to specific domains and are time-consuming ( Ladosz et al. 2022 ) . Indirect methods involve leveraging expert knowledge to guide exploration ( Wang et al. 2020 ; Xu et al. 2021 ) and developing high-quality user simulators to generate diverse experiences ( Peng et al. 2018 ) . However, these approaches add costs for constructing expert knowledge and user simulators. Evolutionary Algorithms (EAs) are theoretically well-suited for balancing exploration and exploitation due to their population-based diversity exploration advantages ( Crepinsek et al. 2013 ) . However, EAs struggle with exploitation because they lack gradient guidance, while gradient descent is a key strength of sample-efficient DRL approaches ( Li et al. 2024 ) . A natural solution is to combine the complementary strengths of EAs and DRL in task-oriented dialogue policy tasks, referred to as ERL-based DP, to achieve an effective balance between exploration and exploitation ( Khadka and Tumer 2018 ; Suri 2022 ) . However, this integration is not straightforward. The inherent flexibility of natural language in TOD tasks results in a vast search space, requiring significant time for EAs to evolve effective strategies(as demonstrated in subsubsection 4.4.1 ). To address the above challenges, we propose an Elite Individual Injection (EII) mechanism, integrating it into an ERL-based DP algorithm to form a novel hybrid algorithm called EIERL. As shown in Figure 1 , EIERL comprises two components: Exploration (left) and Exploitation (right). In the Exploration phase, the EA population generates diverse strategies through selection, crossover, and mutation to explore various experiences, while the EII mechanism periodically injects best-performing individuals into the EA population to accelerate evolution. The injection timing is adaptively controlled by an elite discriminator, which adaptively selects optimal injection moments and updates based on the training stage. In the Exploitation phase, the DRL agent optimizes its performance using exsiting experiences and replicates optimized agents to form a DRL population. Both the formed EA and DRL populations then interact with the environment to generate diverse exploration experiences and exploitation experiences. The strengths and generalizability of EIERL in effectively trade-off exploration and exploitation are demonstrated on four public dialogue datasets. Moreover, the EII mechanism has proven effective in accelerating the EA’s exploration process, enabling a seamless integration of DRL and EA for task-oriented dialogue policy tasks. To our knowledge, this is the first study to integrate EAs and DRL into task-oriented dialogue policy tasks. In summary, our contributions are threefold: • We propose a hybrid algorithm that leverages the complementary strengths of EAs and DRL to effectively balance exploration and exploitation in dialogue policy learning. • We introduce an elite individual injection mechanism to solve the prolonged evolution challenges faced by traditional ERL directly migrating to dialogue task, enabling an effective combination of EA and DRL. • We validate the effectiveness and generalizability of EIERL in multiple dialogue datasets. 2 Related Work 2.1 The Exploration-Exploitation Trade-off Methods for balancing exploration and exploitation in DRL-based task-oriented dialogue policies are generally categorized into direct and indirect enhancements of exploration efficiency. Direct methods involve designing tailored exploration strategies, which are often labor-intensive ( Ladosz et al. 2022 ) . For instance, Tegho et al. 2017 and Lipton et al. 2016 proposed a Bayes-by-Backprop approach based on Deep Q-Networks (DQN), which customizes exploration strategies by utilizing Thompson sampling to estimate uncertainty from Monte Carlo samples. However, the effectiveness of this method heavily relies on the choice of prior distributions, which may limit its applicability across different scenarios. Indirect methods utilize expert knowledge to guide exploration ( Xu et al. 2021 ; Wang et al. 2020 ) or develop high-quality user simulators to generate diverse experiences ( Peng et al. 2018 ) . However, expert-driven approaches depend heavily on data quality, where poor data can hinder exploration or lead to ineffective reward structures ( Kang et al. 2018 ) . For example, Nishimoto and Costa 2019 combined a softmax strategy with transfer learning, using the source task as expert knowledge, and employed a Boltzmann distribution to assign probabilities to action-state pairs, thereby improving exploration effectiveness. Nevertheless, the differences between the source and target tasks in transfer learning restrict the model’s adaptability. For user simulator-based methods, although Peng et al. 2018 demonstrated that user simulators can enhance efficiency, they require

Read the complete text