Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation
Zongxia Li Yapei Chang Yuhang Zhou Xiyang Wu Zichao Liang · 2025
Read Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation by Zongxia Li Yapei Chang Yuhang Zhou Xiyang Wu Zichao Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15068.txt.
Introduction
Zongxia Li and co-authors present in 2025 PrefBERT, a score model for long free form text made by language models. Judging open ended long text is hard, since good and bad work split on sense, flow, and fit. Old scores miss those traits or lean on skewed training data.
PrefBERT learns from two rated data sets with split long styles and graded marks. It then feeds split reward and penalty signals to a group based training loop. Human marks show models trained with its signal read more in line with what people like. It beat two classic text scores in the tests.
Catalog record
| CREATOR | Zongxia Li Yapei Chang Yuhang Zhou Xiyang Wu Zichao Liang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15068
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation Zongxia Li Yapei Chang Yuhang Zhou Xiyang Wu Zichao Liang Abstract Evaluating open-ended long-form generation is challenging because it is hard to define what clearly separates good from bad outputs. Existing methods often miss key aspects like coherence, style, or relevance, or are biased by pretraining data, making open-ended long-form evaluation an underexplored problem. To address this gap, we propose Pref bert , a scoring model for evaluating open-ended long-form generation in grpo and guiding its training with distinct rewards for good and bad outputs. Trained on two response evaluation datasets with diverse long-form styles and Likert-rated quality, Pref bert effectively supports grpo by offering better semantic reward feedback than traditional metrics rouge -L and berts core do. Through comprehensive evaluations, including llm -as-a-judge, human ratings, and qualitative analysis, we show that Pref bert , trained on multi-sentence and paragraph-length responses, remains reliable across varied long passages and aligns well with the verifiable rewards grpo needs. Human evaluations confirm that using Pref bert as the reward signal to train policy models yields responses better aligned with human preferences than those trained with traditional metrics. Our code is available at https://github.com/zli12321/long_form_rl . 1 Introduction Figure 1: Our proposed reward model Pref bert better reflects human judgments by distinguishing the good responses from the bad ones than other traditional metrics, generalized reward model. Identifying the good and bad generations is the key to the success of rlvr to improve llm s’ abilities on structured rule-based tasks such as mathematical problem-solving Ahn et al. 2024 , classification Rouzegar and Makrehchi 2024 , and game planning Shao et al. 2019 . rl algorithms like Group Relative Policy Optimization ( grpo ) excel in mathematical problem-solving tasks by leveraging clear, rule-based reward signals (e.g., correctness) that effectively distinguish between correct and incorrect responses DeepSeek-AI et al. 2025 . However, despite these advances, extending grpo to open-ended, long-form text generation tasks, such as writing a poem, creative compositions, instruction-following, remains largely underexplored. A key challenge in applying grpo to open-ended generation is the lack of objective evaluation criteria, making it difficult to distinguish high-quality responses from poor ones. Krishna et al. 2021 . Unlike structured tasks evaluated by explicit correctness criteria, long-form generation requires models to balance coherence, fluency, and informativeness which are all subjective dimensions that are hard to define with discrete rules. This makes reward design a fundamental challenge in grpo : How do we guide a model to write better, not just longer? A key barrier is the lack of reliable, scalable evaluation methods. Traditional metrics like rouge Lin 2004 and berts core Zhang et al. 2020 —which measure lexical or embedding-level similarity to a reference—often fail to capture qualities that humans actually care about, such as clarity, relevance, and stylistic appropriateness Chen et al. 2019 ; Li et al. 2024 . Thus, traditional metrics correlate poorly with human preferences, making them unreliable reward signals in grpo pipelines. Recent work has applied general reward models ( grm ) Lambert et al. 2024 in rlhf to select preferred responses, but they are not designed as verifiable rewards for grpo , and their large size introduces latency and high computational cost. As shown in Figure 1 , traditional metrics fail to distinguish between clearly better and worse responses, assigning similar scores to both, while grm rewards often correlate with length rather than content quality. To address open-ended evaluation in grpo , we propose Pref bert —a lightweight Modern bert Warner et al. 2024 model trained on diverse long-form responses and human ratings to provide fine-grained quality evaluation. We hypothesize that Pref bert , by capturing human preferences, quality, and semantics, provides stronger reward signals than traditional metrics, enabling grpo to train models that generate higher-quality responses. Through extensive evaluations—including llm -as-a-judge point-wise scoring, pairwise Bradley-Terry ranking analyses Bradley and Terry 1952 , and human rating and qualitative evaluation—we show that leveraging stronger, Pref bert significantly enhances the quality of open-ended text generation across three long-form datasets– eli5 Fan et al. 2019 , Alpaca Taori et al. 2023b , LongForm Köksal et al. 2023 . Our results show that using improved fine-grained evaluators as reward signals in grpo leads to better alignment with human preferences for open-ended response generation compared to traditional metric-based rewardsand. Furthermore, smaller models (e.g., Qwen-2.5-3B-Instruct Qwen et al. 2025 ) trained with our enhanced reward models generate similarly preferred and concise responses as their larger counterparts (e.g., Qwen-2.5-32/72B-Instruct), and outperform models trained with traditional supervised fine-tuning ( sft ) in preference quality. Our contributions are: • We introduce Pref bert , a lightweight long-form reward model that can be easily extended to grpo training. • We validate using Pref bert in grpo to train models on across multiple open-ended generation benchmarks ( eli5 , Alpaca, LongForm), showing resulting model have an overall higher alignment with human preferences compared to traditional metrics and sft training. • Through human expert annotations, we further confirm that models trained with Pref bert align better with human preferences than traditional metrics as rewards, showing a promising direction for using grpo to improve open-ended generation. 2 Related Work rl for llm alignment: rl is pivotal in aligning llm s with human preferences by optimizing non-differentiable objectives, making it valuable for tasks like dialogue Li et al. 2016 , summarization Roit et al. 2023 , and code generation Le et al. 2022 . Popular rlhf methods include dpo , which applies a classification loss over preference data, and ppo , which trains a reward model to guide generation Wu et al. 2023 . However, both of them require substantial human-annotated data or computational resources. To address this, grpo DeepSeek-AI et al. 2025 leverages self-generated data and simple, verifiable reward functions to reduce annotation needs, especially for tasks with clear correctness signals like math Liu et al. 2025 . Extensions such as dapo Yu et al. 2025 , grpo - lead Zhang and Zuo 2025 and DISCO Zhou et al. 2025b broaden grpo ’s capabilities in math problem solving. However, these approaches still rely on rule-based reward designs, leaving their application to open-ended, long-form generation tasks—where there is not a fixed correct response—underexplored. Free-form and open-ended evaluation: Evaluating long-form and open-ended generation in llm s remains difficult Krishna et al. 2021 ; Chen et al. 2019 . Unlike short-form tasks with clear correctness signals, long-form outputs, like summaries, dialogues, or open-ended answers, lack binary ground truths and require assessing coherence, factuality, structure, and helpfulness Chiang et al. 2024 ; Fabbri et al. 2021 ; Li et al. 2025a . Traditional metrics such as rouge Lin 2004 , bleu Papineni et al. 2002 , and berts core Zhang et al. 2020 rely on token overlap or embeddings but poorly reflect semantic or pragmatic qualities, often misaligning with human judgments Chen et al. 2019 . To overcome this, llm -as-a-judge offers more nuanced evaluation through pairwise comparisons or Likert ratings, aligning better with human preferences Chiang et al. 2024 ; Gu et al. 2025 ; Zheng et al. 2023 ; Zhou et al. 2025a . However, this approach introduces heavy computational costs, especially in grpo where multiple generations and evaluations per prompt are