ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning

Jianghao Chen Wei Sun Qixiang Yin Lingxing Kong Zhixing Tan Jiajun Zhang · 2025

Read ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning by Jianghao Chen Wei Sun Qixiang Yin Lingxing Kong Zhixing Tan Jiajun Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04903.txt.

Introduction

Long answers from language models are hard to train for because good long-form data are scarce and scoring is vague. Jianghao Chen, lead author with four co-authors, propose ACE-RL in a 2025 paper. Their framework turns subjective quality into fine-grained constraint checking for reinforcement learning.

Each instruction is first broken into adaptive constraint criteria across key dimensions of long-form writing. A reward mechanism then scores responses by how many constraints they satisfy. Reinforcement learning optimises the model on those signals. On a writing benchmark, the method beats supervised and reinforcement baselines and, in the authors' tests, surpasses a leading proprietary system.

Catalog record

CREATORJianghao Chen Wei Sun Qixiang Yin Lingxing Kong Zhixing Tan Jiajun Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-04903

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning Jianghao Chen Wei Sun Qixiang Yin Lingxing Kong Zhixing Tan Jiajun Zhang Abstract Large Language Models (LLMs) have demonstrated remarkable progress in long-context understanding, yet they face significant challenges in high-quality long-form generation. Existing studies primarily suffer from two limitations: (1) A heavy reliance on scarce, high-quality long-form response data for supervised fine-tuning (SFT) or for pairwise preference reward in reinforcement learning (RL). (2) Focus on coarse-grained quality optimization dimensions, such as relevance, coherence, and helpfulness, overlooking the fine-grained specifics inherent to diverse long-form generation scenarios. To address this issue, we propose a framework using A daptive C onstraint- E nhanced reward for long-form generation R einforcement L earning (ACE-RL). ACE-RL first automatically deconstructs each instruction into a set of fine-grained, adaptive constraint criteria by identifying its underlying intents and demands. Subsequently, we design a reward mechanism that quantifies the quality of long-form responses based on their satisfaction over corresponding constraints, converting subjective quality evaluation into constraint verification. Finally, we utilize reinforcement learning to guide models toward superior long-form generation capabilities. Experimental results demonstrate that our ACE-RL framework significantly outperforms existing SFT and RL baselines by 20.70% and 7.32% on WritingBench, and our top-performing model even surpasses proprietary systems like GPT-4o by 7.10%, providing a more effective training paradigm for LLMs to generate high-quality content across diverse long-form generation scenarios. 1 1 1 Our code is available at https://github.com/ZNLP/ACE-RL 1 Introduction Recent advancements in Large Language Models (LLMs) have yielded significant improvements in their long-context understanding capabilities ( Google 2025 ; OpenAI 2024a ; Anthropic 2025 ; Yang et al. 2025a ) . This success has naturally pushed the research frontier towards a more challenging field: high-quality long-form generation (e.g., storytelling, report writing, or legal document drafting). However, generating coherent, well-structured, and engaging content over thousands of words still remains a significant and unresolved challenge for current long-context LLMs ( Wu et al. 2025b ; Bai et al. 2025 ; Wu et al. 2025c ) . Figure 1: Comparison of reward mechanisms for long-form generation: conventional methods vs. our proposed method. While many researchers focus on supervised fine-tuning (SFT) ( Bai et al. 2025 ; Pham, Sun, and Iyyer 2024 ) , these approaches are often constrained by reliance on costly synthetic datasets from proprietary LLMs and a lack of optimization for response quality attributes ( Deng, Kuleshov, and Rush 2022 ; Pham, Sun, and Iyyer 2024 ) . In contrast, reinforcement learning (RL) alleviates this by enabling direct optimization of desired qualities through a reward mechanism. This allows for a more flexible and robust approach to training models for long-form generation, particularly when focusing on specific human preferences. However, existing RL-based methods primarily rely on pairwise preference reward with coarse-grained evaluation dimensions ( Lei et al. 2025 ; Wu et al. 2025a ) . As shown in Figure 1 , while these methods ensure fundamental qualities such as relevance, coherence, and helpfulness, they fall short of addressing the detailed, instruction-adaptive demands of diverse tasks. For example, given a storytelling instruction: “Write a 3000-word novel about a young, struggling couple with an O. Henry-style ending." The core of the instruction is the writing style of a surprise, ironic twist, a quality that is not a universal marker of “good" writing, but a highly specific stylistic demand. Moreover, the reliance on extra preference response pairs imposes a significant data collection burden, making the process both costly and difficult to scale. To address the above issue, we propose a framework with A daptive C onstraint- E nhanced reward for long-form generation R einforcement L earning (ACE-RL). The core innovation of ACE-RL lies in its training paradigm shift away from coarse-grained, subjective rewards toward fine-grained, verifiable rewards. Specifically, our framework begins by employing an automated pipeline to collect and analyze diverse long-form generation instructions from real-world human-LLM interactions. This pipeline deconstructs each instruction by identifying both its explicit demands and implicit intents, converting them into a checklist of verifiable constraints. We then introduce a novel reward mechanism that scores a generated response based on its satisfaction over these constraints. This process effectively transforms the subjective challenge of evaluating writing quality into a series of constraint verification tasks. Finally, by leveraging reinforcement learning guided by this fine-grained reward signal, ACE-RL significantly enhances the long-form generation capability of LLMs. Our contributions are summarized as follows: • We propose ACE-RL, a novel reinforcement learning framework that utilizes fine-grained, instruction-adaptive rewards for long-form generation. • We introduce an automatic pipeline for constructing long-form generation instructions with constraint checklists, and we will release our datasets for future studies. • We design a novel reward mechanism that transforms subjective quality evaluation of long-form responses into constraint verification tasks. • Extensive experimental results demonstrate that our method significantly outperforms SFT and RL baselines, and even surpasses strong proprietary LLMs. 2 Related Work 2.1 Long-context Understanding for LLMs The enhanced long-context understanding capabilities of LLMs have significantly broadened their application across various fields, such as long-chain reasoning ( Yeo et al. 2025 ; Chen et al. 2025a ; Yang, Liao, and Fan 2025 ) , long video and image processing ( Guo et al. 2025b ; Jian et al. 2025 ; Jian, Yu, and Zhang 2024 ) , and agent interaction ( Zhang et al. 2024 ; Ren et al. 2025 ; Xu et al. 2025 ) . Recent studies have proposed both model-centric and data-centric methods. Model-centric methods involve novel model architectures ( Gu and Dao 2024 ; Dao and Gu 2024 ; Peng et al. 2023 ) , efficient attention ( Liu et al. 2024 ; Lu et al. 2025 ; Shah et al. 2024 ) , and extrapolation of positional encoding ( Xiong et al. 2024 ; Peng et al. 2024 ; Ding et al. 2024 ; An et al. 2024a ) . These approaches focus on redesigning the core components of LLMs to handle extensive contexts more efficiently and effectively. In parallel, data-centric methods improve the long-context ability of LLMs through continual pre-training ( Fu et al. 2024 ; Chen et al. 2024a ; Chen et al. 2025b ) or supervised fine-tuning ( Bai et al. 2024 ; An et al. 2024b ; Chen et al. 2024b ) . However, these studies mainly focus on long-context input scenarios, and generating high-quality long-form outputs remains a significant challenge for current LLMs. 2.2 Long-form Generation for LLMs The challenge of long-form generation has been approached primarily from two directions: supervised fine-tuning and reinforcement learning. SFT-based methods focus on data synthesis, such as long-form response construction ( Quan et al. 2024 ; Bai et al. 2025 ) and instruction back-translation ( Pham, Sun, and Iyyer 2024 ) . Recent success of reinforcement learning with preference reward ( Sun, Shen, and Ton 2025 ; Yang et al. 2025b ) encourages researchers to explore outcome-based RL for long-form generation. Lei et al. 2025 leverage proprietary LLMs to generate reference responses and calculate pairwise reward by quality comparison of model responses and reference responses. Wu et al. 2025a utilize a reward model trained with preference data pairs to capture general response quality such as fluency, coherence, and helpfulness. However, these approaches rely on coarse-grained

Read the complete text