RLMR: Reinforcement Learning with Mixed Rewards for Creative Writing

Jianxing Liao, Tian Zhang, Xiao Feng, Yusong Zhang, Rui Yang, Haorui Wang, Bosi Wen, Ziying Wang, Runzhi Shi · 2025

Read RLMR: Reinforcement Learning with Mixed Rewards for Creative Writing by Jianxing Liao, Tian Zhang, Xiao Feng, Yusong Zhang, Rui Yang, Haorui Wang, Bosi Wen, Ziying Wang, Runzhi Shi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18642.txt.

Introduction

Jianxing Liao and eight co-authors teach language models to write creatively while obeying the rules of the assignment. Good creative writing must balance flair, such as vivid style and feeling, against hard constraints like format and word limits. Single-score training improves one side at the expense of the other, and fixed blends cannot adapt to different tasks. Their method, RLMR, mixes two rewards whose balance shifts during training.

One reward grades writing quality through a trained judge model. The other checks constraint following by direct verification. The constraint weight rises or falls with the quality spread inside each batch of samples, so rule-breaking samples earn negative credit and get pushed down. Across models from eight to seventy-two billion parameters, the method improved both rule following and writing quality. Human experts preferred its writing in most paired tests on a new real-world benchmark the team built.

Catalog record

CREATORJianxing Liao, Tian Zhang, Xiao Feng, Yusong Zhang, Rui Yang, Haorui Wang, Bosi Wen, Ziying Wang, Runzhi Shi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18642

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RLMR: Reinforcement Learning with Mixed Rewards for Creative Writing Jianxing Liao, Tian Zhang, Xiao Feng, Yusong Zhang, Rui Yang, Haorui Wang, Bosi Wen, Ziying Wang, Runzhi Shi Abstract Large language models are extensively utilized in creative writing applications. Creative writing requires a balance between subjective writing quality (e.g., literariness and emotional expression) and objective constraint following (e.g., format requirements and word limits). Existing methods find it difficult to balance these two aspects: single reward strategies fail to improve both abilities simultaneously, while fixed-weight mixed-reward methods lack the ability to adapt to different writing scenarios. To address this problem, we propose Reinforcement Learning with Mixed Rewards (RLMR), utilizing a dynamically mixed reward system from a writing reward model evaluating subjective writing quality and a constraint verification model assessing objective constraint following. The constraint following reward weight is adjusted dynamically according to the writing quality within sampled groups, ensuring that samples violating constraints get negative advantage in GRPO and thus penalized during training, which is the key innovation of this proposed method. We conduct automated and manual evaluations across diverse model families from 8B to 72B parameters. Additionally, we construct a real-world writing benchmark named WriteEval for comprehensive evaluation. Results illustrate that our method achieves consistent improvements in both instruction following (IFEval from 83.36% to 86.65%) and writing quality (72.75% win rate in manual expert pairwise evaluations on WriteEval). To the best of our knowledge, RLMR is the first work to combine subjective preferences with objective verification in online RL training, providing an effective solution for multi-dimensional creative writing optimization. Introduction Large language models (LLMs) are widely applied to creative writing tasks, from traditional poetry composition to modern fiction generation, and from literary scriptwriting to commercial copywriting, fulfilling diverse writing demands across domains and genres. To further enhance LLM performance in creative writing tasks, reinforcement learning techniques have been widely applied during the post-training phase. Through methods such as Group Relative Policy Optimization (GRPO), researchers aim to guide models toward generating higher-quality creative content through reward signals. However, existing reinforcement learning reward strategies suffer from fundamental limitations. The evaluation criteria for creative writing are inherently dual in nature: on one hand, they require assessing subjective writing qualities such as literariness, emotional expression, and originality; on the other hand, they necessitate verifying objective constraint following, including length constraints, format requirements, and specific writing styles. Different creative writing scenarios exhibit significant variations in their emphasis on subjective versus objective evaluation. Figure 1: Comparison of single reward strategy versus our mixed RLMR approach. Given a task requiring an advertising slogan starting with ”X” using no more than 15 words, Response A follows constraints but scores lower (8.9), while Response B violates constraints but scores higher (9.0). Single reward strategies incorrectly prefer Response B, while our RLMR combines writing quality and instruction following signals to correctly identify Response A as superior through dynamic penalty adjustments. Current reward strategies face two major challenges. First, single reward strategies struggle to simultaneously optimize both subjective and objective dimensions. As illustrated in Figure 1 , under single-signal strategies, reward models only score writing quality without reflecting constraint following. Second, existing multi-reward signal fusion strategies typically employ fixed-weight summation. Such fixed-weight mechanisms fail to dynamically adjust weights based on actual sample performance within groups, making them unsuitable for different writing scenarios. To address these issues, we propose Reinforcement Learning with Mixed Rewards (RLMR), a dynamic mixed-reward framework for creative writing. By coupling a writing reward model for evaluating subjective writing quality with a constraint verification model for assessing objective constraint following, we implement an adaptive mechanism that dynamically allocates reward weights based on constraint following within sampled group responses. Unlike existing methods that use fixed-weight fusion, our core innovation lies in dynamically adjusting the constraint following reward weight according to writing quality within sampled groups. This ensures that samples violating constraints receive negative advantage values in GRPO calculations, thereby being systematically penalized during policy gradient updates. To validate our method’s effectiveness, we conducted training on various scales of Qwen and DeepSeek model families and performed both automated and manual evaluations on multiple creative writing and instruction-following benchmarks. RLMR shows substantial gains in both writing quality and constraint following compared to single-reward and linear weighting baseline methods. Manual evaluation confirms significant preference for our approach over traditional strategies. These results effectively validate that our method resolves the trade-off between subjective and objective evaluation criteria in creative writing optimization. Our key contributions include: 1. Identifying the inherent limitations of single reward signals and fixed-weight mixing strategies in creative writing tasks. 2. Proposing RLMR and developing a dynamic reward adjustment mechanism that ensures constraint-violating samples receive negative advantages during training, enabling better balance between writing quality and constraint following among multiple reward signals. 3. Demonstrating consistent improvements across diverse model families and scales through comprehensive automated and manual evaluations, proving the effectiveness of our method. Figure 2: Overview of our Dynamic Mixed-Reward GRPO Framework. The policy model generates responses ( o_{1},o_{2},o_{3} ) evaluated by both writing quality (Writing RM) and constraint compliance (Validator). In this example: n=3 (total samples), r_{\max}^{\text{vio}}=8 (highest reward among violating samples), \gamma=1 (minimum gap below the mean), k=1 (number of violating samples), \sum_{i=1}^{n}r_{i}=19 (sum of original rewards). The framework calculates penalty \delta=4 and deducts it from violating samples ( o_{2}:8\rightarrow 4 ). After adjustment around mean=5, only high-quality compliant samples ( o_{1} ) receive positive gradients (green), while both low-quality samples ( o_{3} ) and constraint-violating samples ( o_{2} ) receive negative gradients (red). Related Work To further improve LLM performance and align it with human preferences, reinforcement learning, especially RLHF, has become a mainstream optimization approach. Algorithms such as Proximal Policy Optimization (PPO) ( Ouyang et al. 2022 ) and Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) are widely used to align LLM behavior with human preferences. PPO ensures training stability by limiting the extent of policy updates through clipped probability ratios, but requires separate value function training which increases computational overhead. GRPO optimizes policy gradients by estimating baselines from sampled groups, avoiding the need for separate value function training while maintaining competitive performance. Given GRPO’s computational efficiency and effectiveness in creative writing scenarios, we choose it as our reinforcement learning framework. Mixed reward strategies have become increasingly important in reinforcement learning, integrating multi-dimensional reward signals to guide model training more comprehensively. Peng et al. ( Peng et al. 2025a ) proposed the Agentic Reward Modeling framework, which combines human preference rewards with verifiable correctness signals (factuality and instruction following) to provide more reliable rewards for large language models. Jia et al. ( Jia et al. 2025 ) introduced Writing-Zero, proposing a writing-principle-based pairwise Generative Reward Model (GRM) that leverages self-principled critique to transform subjective assessments into reliable, verifiable rewards for creative writing tasks. Wu et al. ( Wu et al. 2025 ) developed LongWriter-Zero framework for ultra-long text generation, employing specialized reward models targeting length control, writing quality, and structural formatting with a composite reward function that averages individual advantages to balance multiple reward dimensions. However, these existing mixed reward approaches all rely on fixed-weight fusion mechanisms, which suffer from fundamental limitations. First, fixed weights cannot adapt to varying constraint compliance patterns within different

Read the complete text