Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards

Ruipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo · 2025

Read Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards by Ruipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00103.txt.

Introduction

Ruipeng Jia and colleagues teach language models to write well with checkable rewards, in work from 2025. Reward based training shines on math and code, where answers check out true or false. Writing quality is a matter of taste, so fixed score models get gamed by length tricks.

Their pairwise judge grades drafts against each other using stated writing rules. Side by side calls turn taste calls into rewards that resist gaming far better. A self reference training step compares drafts with a fresh peer draft, needing no fixed gold text. The trained writer beat score model baselines and resisted reward hacks in tests. Benchmark scores matched strong rivals on home and open writing sets.

Catalog record

CREATORRuipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00103

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards Ruipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo Abstract Reinforcement learning with verifiable rewards (RLVR) has facilitated significant advances in large language models (LLMs), particularly for reasoning tasks with objective, ground-truth answers, such as math and code generation. However, a substantial gap persists for non-verifiable tasks—such as creative writing and open-ended dialogue—where quality assessment is inherently subjective and lacks definitive, externally verifiable references. Existing methodologies for these tasks predominantly rely on scalar reward models trained using human preferences, but these models suffer from limited generalization and are vulnerable to reward hacking, including issues such as over-explanation and length bias. In this work, we propose a unified RLVR-based training paradigm that effectively bridges the gap between non-verifiable tasks and verifiable reward. We introduce a novel pairwise Generative Reward Model (GenRM) grounded in writing principles and a new Bootstrapped Relative Policy Optimization (BRPO) algorithm. The pairwise writing GenRM applies self-principled critique to transform subjective assessments into robust, verifiable rewards, while BRPO facilitates dynamic, reference-free pairwise comparisons by utilizing bootstrapped responses as temporary references during group rollouts in reinforcement learning (RL) training. Our approach enables LLMs to cultivate advanced writing capabilities without requiring supervised fine-tuning. This is demonstrated by Writing-Zero, which exhibits consistent performance improvements and enhanced resilience to reward hacking compared to scalar reward baselines. In addition, our method achieves competitive results on both proprietary and publicly available writing benchmarks. These results suggest the potential for unifying rule-based, reference-based, and reference-free reward modeling within the RLVR framework, thereby advancing the development of a comprehensive and scalable RL training paradigm with broad applicability across various language tasks. 1 Introduction Recently, large language models (LLMs) have achieved remarkable breakthroughs in reasoning capabilities through reinforcement learning with verifiable rewards (RLVR), particularly in solving complex logical tasks such as mathematics and programming ( Guo et al., 2025 ) . RLVR relies on reference-based signals, where the availability of objective ground-truth answers enables reliable verification of model responses. This approach has proven particularly effective in tasks with well-defined solutions, such as mathematical reasoning and code generation, where simple rule-based verifiers can provide clear binary signals (correct or incorrect) ( Team et al., 2025 ) . However, there is a spectrum ranging from verifiable to non-verifiable problems, with problems like mathematics and coding at one end, multi-subject QA with less structured answers in the middle, and creative writing, multi-turn dialogue that lack a reference answer and require quality assessment based on human preferences at the non-verifiable end. For less-verifiable or non-verifiable problems, previous works mainly rely on a scalar reward model trained with human preference data for RLHF training, which has limited generalization ability and is prone to reward hacking ( Zhong et al., 2025 ) . For example, in creative writing scenarios, models trained with RLHF often exhibit over-explanation, where they append lengthy justifications of how their response perfectly meets user requirements, even when the actual content fails to do so. Figure 1: Comparison of Eval RM scores during Writing-Zero training. The blue line shows Qwen3-32B-Base-ScalarRM-GRPO , while the orange line shows Writing-Zero ( Qwen3-32B-Base-GenRM-BRPO ). The dashed red and green lines indicate the SFT (Writing-SFT) and SFT+RL (Writing-SFT-GenRM-BRPO) baselines, respectively. Developing high-quality and robust reward modeling methods for more general domains remains a challenging and active research direction. Su et al. (2025) propose training a generative reward model to verify whether the response matches the reference answer, expanding the application of RLVR to less structured domains such as multi-subject QA. Most recently, researchers attempt directly adapting RLVR to improve generative reward modeling (GenRM) and train GRM in R1-style with long COT of designed evaluation principles and critiques, which results in more accurate and reliable rewards across broader domains and effective test-time scaling ( Liu et al., 2025 ; Chen et al., 2025a ) . While the emergence of reliable and robust GenRM has opened promising avenues for extending RLVR to broader domains, its effective application, particularly to non-verifiable tasks, remains a challenging and largely unexplored research topic. As the Chinese saying goes, there is no absolute first place in literary arts, creative writing represents one of the typical non-verifiable tasks where quality assessment is inherently subjective and lacks definitive reference. In this work, we aim to bridge the gap between the non-verifiable tasks and verifiable rewards and propose a new training paradigm for non-verifiable writing tasks, with writing-principle-based pairwise GenRM and a new RL algorithm Bootstrapped Relative Policy Optimization (BRPO). Specifically, inspired by Liu et al. (2025) , we first train the Pairwise Writing GenRM through high-quality self-principled critique cold-start data, and perform RLVR to refine the GenRM’s ability to formulate adaptive principles and nuanced critiques specific to different writing scenarios and diverse response pairs, thereby producing more reliable outcome rewards for creative writing evaluation. The Writing-GenRM takes in a pair of responses and outputs two respective scores between 0 and 10 to indicate comparative quality, effectively transforming subjective assessments into reliable verifiable rewards. Then we introduce Bootstrapped Relative Policy Optimization (BRPO), which dynamically selects samples from within the group rollouts as temporary references for pairwise comparison and advantage estimation. This bootstrap mechanism eliminates the need for fixed external references, enabling the policy model to continuously improve by leveraging its own increasingly sophisticated outputs for comparison. Similar to DeepSeek-R1 ( Guo et al., 2025 ) , we explore the potential of LLMs developing writing capabilities without any supervised finetuning data and introduce Writing-Zero using Qwen3-32B-Base ( Yang et al., 2025 ) as the base model. Throughout the training process, Writing-Zero shows consistent improvement and exceptional resistance against reward hacking compared with the base model trained with scalar reward, as shown in Figure 1 . Furthermore, we introduce Writing-R1 using an in-house thinking SFT model as the base model and achieve competitive results on in-house and open-source writing benchmarks. Our empirical results demonstrate the effectiveness of our proposed method. To sum up, our work represents an important step in unifying different reward modeling paradigms under the RLVR training framework. We demonstrate that by leveraging pairwise generative reward modeling with self-principled critique, even non-verifiable tasks like creative writing can benefit from stable and scalable RL training, similar to verifiable tasks. This opens up the possibility of unifying three major reward modeling approaches: rule-based rewards for well-defined tasks, model-based rewards with reference answers for less-structured tasks, and model-based rewards without reference answers for creative tasks. Our work thus paves the way for establishing a comprehensive and consistent RLVR training paradigm that can be applied across the entire spectrum of language tasks, from verifiable to non-verifiable domains. 2 Preliminary 2.1 Group Relative Policy Optimization GRPO ( Shao et al., 2024 ) is an efficient and effective RL algorithm, by introducing the group-relative normalization to estimate the advantage and eliminating the value function of PPO ( Schulman et al., 2017 ) . For a specific question q in dataset \mathcal{D} , the behavior policy \pi_{\theta_{old}} samples a group of G individual responses \{o_{i}\}_{i=1}^{G} . Then, the advantage of the i -th response is calculated by normalizing the group-level rewards \{R_{i}\}_{i=1}^{G} . Like PPO, GRPO also stabilizes training and improves sample efficiency by constraining the policy model \pi_{\theta} updates within a proximal

Read the complete text