CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization Corresponding Author

Xinge Ye, Rui Wang, Yuchuan Wu, Victor Ma, Feiteng Fang, Fei Huang, Yongbin Li · 2025

Read CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization Corresponding Author by Xinge Ye, Rui Wang, Yuchuan Wu, Victor Ma, Feiteng Fang, Fei Huang, Yongbin Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-09074.txt.

Introduction

Xinge Ye and co-authors fix reward design for role-playing chatbots in a 2025 paper. Tuning by reward works well where answers can be checked, like code or math. Role-play has no single right reply, so sample-by-sample scores wobble and blur.

Their method, Comparative Policy Optimization, grades replies as a group, not one by one. Human judging works that way too, mixing stated rules with side-by-side feel. A companion test arena, CharacterArena, judges characters on the same comparative plan. Steadier group rewards should yield chatbots that stay in character longer. The work targets a weak point in training open-ended talk.

Catalog record

CREATORXinge Ye, Rui Wang, Yuchuan Wu, Victor Ma, Feiteng Fang, Fei Huang, Yongbin Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-09074

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization Corresponding Author Xinge Ye, Rui Wang, Yuchuan Wu, Victor Ma, Feiteng Fang, Fei Huang, Yongbin Li Abstract Reinforcement Learning Fine-Tuning (RLFT) has achieved notable success in tasks with objectively verifiable answers (e.g., code generation, mathematical reasoning), yet struggles with open-ended subjective tasks like role-playing dialogue. Traditional reward modeling approaches, which rely on independent sample-wise scoring, face dual challenges: subjective evaluation criteria and unstable reward signals. Motivated by the insight that human evaluation inherently combines explicit criteria with implicit comparative judgments, we propose Comparative Policy Optimization (CPO) . CPO redefines the reward evaluation paradigm by shifting from sample-wise scoring to comparative group-wise scoring. Building on the same principle, we introduce the CharacterArena evaluation framework, which comprises two stages: (1) Contextualized Multi-turn Role-playing Simulation , and (2) Trajectory-level Comparative Evaluation . By operationalizing subjective scoring via objective trajectory comparisons, CharacterArena minimizes contextual bias and enables more robust and fair performance evaluation. Empirical results on CharacterEval, CharacterBench, and CharacterArena confirm that CPO effectively mitigates reward ambiguity and leads to substantial improvements in dialogue quality. 1 Introduction Figure 1: Challenges in role-playing reward estimation. Sample-wise LLM evaluation suffers from ambiguous criteria, prompt sensitivity, and scoring instability, amplifying errors in GRPO’s advantage computation. Group-wise rewarding provides clearer evaluation criteria and implicit comparisons. Additionally, ranking-based evaluation is simpler and more stable than independent scoring, reducing error propagation in advantage estimation. Role-playing dialogue systems aim to support immersive multi-turn interactions by simulating specific character personas Zhou et al. 2024b ; Wang et al. 2024c . A core challenge lies in generating responses that are not only coherent with the character profile but also rich in narrative appeal and stylistic diversity Zhou et al. 2024a . Existing approaches primarily adopt Supervised Fine-Tuning (SFT) Chung et al. 2022 over high-quality dialogue corpora Wang et al. 2024a ; Wang et al. 2025b , following an imitation learning paradigm. While SFT has proven effective to a degree, it is inherently constrained by the distribution of the training data—leading to overfitting on observed dialogue patterns, limited creative flexibility, and often underwhelming performance in delivering truly engaging and immersive role-playing experiences Sun and van der Schaar 2025 . Recent advances in Reinforcement Learning Fine-Tuning (RLFT) Christiano et al. 2017 ; Lee et al. 2024 ; Rafailov et al. 2023 present a promising alternative. RLFT has achieved notable success as a post-training strategy for large language models (LLMs), particularly in aligning outputs with human preferences Achiam et al. 2023 and enhancing complex reasoning abilities DeepSeek-AI et al. 2025 ; Qwen et al. 2025 ; Yang et al. 2025 . However, the effectiveness of RLFT hinges on the availability of reliable and discriminative reward signals — a condition that holds in well-structured domains with objective correctness criteria Liu et al. 2025 . In stark contrast, role-playing poses a fundamentally more subjective and open-ended challenge, where key objectives like personality consistency, narrative appeal, and emotional resonance are inherently ill-defined. As illustrated in Figure 1 , current reward modeling methods encounter two limitations. First, the evaluation criteria for open-ended responses are intrinsically ambiguous, making it difficult to establish reliable and consistent scoring rules. Second, existing sample-wise LLM-based evaluators are highly sensitive to prompt variations, often producing unstable and weakly discriminative scores, and in some cases, collapsing most outputs into a narrow scoring range Yuan et al. 2024a . To address these issues and unlock the potential of RLFT in role-playing, we propose Comparative Policy Optimization (CPO) . Inspired by the observation that human evaluation relies not only on explicit criteria but also on implicit comparisons between samples Yuan et al. 2024a , CPO reframes reward modeling from individual sample scoring to group-wise scoring. This shift leads to an approximately 20% improvement in human agreement, effectively reducing reward ambiguity. Furthermore, we introduce CharacterArena , a comprehensive evaluation framework. It comprises two stages: (1) Contextualized Multi-turn Role-playing Simulation , which generates interaction trajectories under controlled character and scenario settings; and (2) Trajectory-based Comparative Evaluation , which enables fair and robust assessments by anchoring evaluations to direct trajectory comparisons rather than absolute scores. Experimental results demonstrate that CPO consistently outperforms other RLFT methods on both CharacterEval and CharacterBench, and surpasses all baselines within the CharacterArena evaluation framework. These findings highlight the effectiveness of our approach in addressing reward ambiguity and substantially enhancing dialogue quality in role-playing systems. Our main contributions are as follows: • We propose Comparative Policy Optimization (CPO), a new RLFT method based on a group-wise reward modeling paradigm, specifically designed to reduce reward ambiguity in open-ended role-playing dialogues. • We present CharacterArena, a new evaluation framework that transforms subjective judgments into more objective comparisons, effectively minimizing contextual bias and enabling a fairer assessment of role-playing performance. • Experimental results on three benchmarks show that CPO outperforms existing RLHF methods. Further analysis demonstrates that group-wise scoring improves human agreement by up to 20%. 2 Related Work 2.1 Role-playing The development of Role-Playing Agents has been revolutionized by recent breakthroughs in LLMs Dubey et al. 2024 ; Qwen et al. 2025 ; Achiam et al. 2023 , empowering users with unprecedented freedom to create and customize their own characters for engaging interactions Li et al. 2024 ; Chen et al. 2023 ; Wang et al. 2024a ; Lu et al. 2024 ; Chen et al. 2024 . This customization often relies on two approaches: prompting general-purpose LLMs for role-play Wang et al. 2025a , or developing specialized LLMs specifically tailored for character generation by training them on role-playing dialogues Zhou et al. 2024a ; Xu et al. 2024 ; Wang et al. 2025b . Current evaluation of Role-Playing Agents (RPAs) employs two primary paradigms: question-answering (QA) and LLM-based judging benchmarks. QA benchmarks typically use multiple-choice questions to target specific RPA capabilities, such as character knowledge Shen et al. 2024 , decision-making Xu et al. 2024 , motivation recognition Yuan et al. 2024b , and personality fidelity Shao et al. 2023 ; Wang et al. 2024b . Conversely, the LLM-based judging approach prompts RPAs with predefined questions to assess role-playing performance, which is then scored by LLM judges or reward models Tu et al. 2024 ; Zhou et al. 2024b ; Wang et al. 2025a ; Dai et al. 2025 . This method generally allows for a more comprehensive evaluation, assessing conversational skills, character adherence (knowledge and personality), and interaction engagement. A significant limitation of LLM-judging, however, is its dependence on static, externally provided conversation histories. This overlooks the vital multi-turn dynamics of interactive role-playing and risks introducing biases from context not generated by the agent under evaluation Wang et al. 2025b . 2.2 Reinforcement Learning Fine-tuning Reinforcement Learning Fine-tuning (RLFT) has become a widely adopted approach for post-training LLMs at scale, significantly enhancing their emergent capabilities Yuan et al. 2023 ; Rafailov et al. 2023 ; Christiano et al. 2017 . A key challenge of RL is to obtain accurate reward signals for LLMs in specific domains. Recent studies on RLFT primarily focus on highly structured tasks with well-defined rules and verifiable ground-truth answers, such as code generation and mathematical reasoning Yeo et al. 2025 ; Pan et al. 2025 ; Zeng et al. 2025 ; Cui et

Read the complete text