Learning to Reason for Long-Form Story Generation

Alexander Gurung, Mirella Lapata · 2025

Read Learning to Reason for Long-Form Story Generation by Alexander Gurung, Mirella Lapata (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22828.txt.

Introduction

Alexander Gurung and Mirella Lapata teach a model to reason for long-form story generation in a 2025 paper. A good long tale needs plot and character arcs held in line for thousands of words, in a style that stays bright. Labeled data and clean scores for that art are scarce, so past work leans on hand-built prompts.

Their plan makes the next chapter of a book the task, and pays the model when its guess of the true text gets better in a checked way. Books with no labels then serve as the training signal. Reasoning for the tale is learned, not prompted by hand.

Catalog record

CREATORAlexander Gurung, Mirella Lapata
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22828

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning to Reason for Long-Form Story Generation Alexander Gurung, Mirella Lapata Abstract Generating high-quality stories spanning thousands of tokens requires competency across a variety of skills, from tracking plot and character arcs to keeping a consistent and engaging style. Due to the difficulty of sourcing labeled datasets and precise quality measurements, most work using large language models (LLMs) for long-form story generation resorts to combinations of hand-designed prompting techniques to elicit author-like behavior. This is a manual process that is highly dependent on the specific story-generation task. Motivated by the recent success of applying RL with Verifiable Rewards to domains like math and coding, we propose a general story-generation task (Next-Chapter Prediction) and a reward formulation (Verifiable Rewards via Completion Likelihood Improvement) that allows us to use an unlabeled book dataset as a learning signal for reasoning. We learn to reason over a story’s condensed information and generate a detailed plan for the next chapter. Our reasoning is evaluated via the chapters it helps a story generator create, and compared against non-trained and supervised fine-tuning (SFT) baselines. Pairwise human judgments reveal the chapters our learned reasoning produces are preferred across almost all metrics, and the effect is more pronounced in Sci-Fi and Fantasy genres. 1 1 1 We release reproduction and training code at github.com/Alex-Gurung/ReasoningNCP . 1 Introduction Long-form story generation is a difficult modeling task that requires synthesizing thousands of tokens of rich and subtle text into coherent and character-aware narratives. High-quality writing balances interesting characters with engaging world-building and satisfying plot arcs ( Jarvis, 2014 ; Kyle, 2016 ) , posing problems of long-term dependencies and accurate Theory-of-Mind modeling. Although Large Language Models (LLMs) have shown recent promise in writing long texts ( Yang et al., 2022 ; Yang et al., 2023 ; Bai et al., 2025 ; Xie et al., 2023 ; Shao et al., 2024a ) , their stories still struggle on a variety of criteria like originality, plot, character development, and pacing ( Chakrabarty et al., 2024 ; Wang et al., 2023 ; Wang et al., 2022 ; Ismayilzada et al., 2024 ) , in addition to more fundamental long-form generation flaws like repetition and quality degradation ( Que et al., 2024 ; Wu et al., 2024 ) . Outside of long-form story generation, Reinforcement Learning (RL) has been successfully applied to post-training LLMs across various tasks, from optimizing general human preferences to improving performance in specialized domains like code generation and math ( Li et al., 2025 ; Lambert et al., 2024 ; Kumar et al., 2025 ) . Although some RL work briefly touches on creative writing, these efforts typically focus on short-form tasks and are framed as an overall preference alignment ( Nguyen et al., 2024 ; Zhao et al., 2024 ) . In general, RL methods for LLMs have necessitated at least one of the following: (a) high quality datasets labeled with preferences, rewards, or verified reasoning traces or (b) high-quality reward models , often in the form of verifiable reward functions. Large-scale datasets have yielded strong results for broader tasks like human-preference-alignment ( Rafailov et al., 2023 ; Ethayarajh et al., 2024 ; Cui et al., 2024 ; Wang et al., 2025a ) , while recent efforts in the math and coding domains have achieved significant success through the use of high-quality rewards ( Gehring et al., 2025 ; DeepSeek-AI et al., 2025 ; Shao et al., 2024b ; Lambert et al., 2024 ; Kimi Team et al., 2025 ; Li et al., 2025 ; Kumar et al., 2025 ) . In particular, there has been a recent surge of interest in enhancing the reasoning capabilities of LLMs using verifiable rewards ( Lambert et al., 2024 ) . Instead of relying on a learned reward model, recent work uses deterministic verification functions — usually rule-based methods that assess answers for correctness and formatting. Neither approach is practical for long-form story generation: story ‘correctness’ is ill-defined, and collecting large datasets of labeled story completions is challenging. The multifaceted and nuanced nature of narratives also makes the formulation of singular reward functions difficult. Instead, the ‘quality’ of stories or their continuation is usually assessed through pairwise comparisons across multiple metrics ( Yang et al., 2022 ; Huot et al., 2025 ; Yang et al., 2023 ; Xie & Riedl, 2024 ) . Most current work on story generation has therefore avoided RL, and instead leaned into hand-designed systems that mimic different parts of the human writing process, like drafting, editing, and planning. ( Fan et al., 2018 ; Zhou et al., 2023 ; Wang et al., 2023 ; Xie & Riedl, 2024 ; Wang et al., 2025b ) . Huot et al. (2025) train LLMs to model human story preferences across metrics like creativity and language use, but they do not train story generation models using these evaluators. Rather than predicting and evaluating entire book-length generations, we propose Next-Chapter Prediction (NCP) as a more tractable and informative task. Inspired by the human book-writing process ( Kyle, 2016 ; Jarvis, 2014 ) that uses both a high-level sketch of the story and more fine-grained information about characters and plot progression, we model story generation as predicting the next chapter given a similar collection of story information. Fine-tuning directly on this task, quickly leads to overfitting and fails to capture the underlying reasoning behind the story-writing process. Instead, we propose a novel reasoning paradigm akin to Reinforcement Learning from Verifiable Rewards (RLVR; Lambert et al. 2024 ) that allows RL training of reasoning traces for story generation. (a) NCP GRPO Training (b) VR-CLI Reward Figure 1 : Next-Chapter Prediction (NCP) training procedure (a) using our VR-CLI reward paradigm (b), with GRPO ( Shao et al., 2024b ) . Our reward uses a reference model to get the improved likelihood of the true next chapter. The reference model is a copy of the policy model frozen at the start of training, used both as our story generator \pi^{\mathcal{G}} and for computing KL-divergence. The policy model is our reasoning model \pi^{\mathcal{R}}_{\theta} , trained to produce detailed plans of the next chapter. VR-CLI is described in Section 6 and training in Section 5 . We introduce Verifiable Rewards via Completion Likelihood Improvement (VR-CLI), a reward modeling paradigm designed to learn reasoning traces that enhance a generator’s ability to reproduce a given dataset (see Figure 1 ). Our key assumption is that increasing the generator’s likelihood of predicting the next chapter will, in turn, improve the quality of its generations. Accordingly, our reward is defined with respect to the improvement in predicting a gold next chapter, which can be naturally expressed in terms of per-token perplexity. We use VR-CLI to train Qwen 2.5 models ( Qwen et al., 2024 ) to produce traces that cite and reason about the given story information before constructing a detailed plan, which is then fed as context to the base model for generating the chapter continuation. Our results show that these models outperform reasoning and non-reasoning baselines and models trained via SFT on next-chapter prediction. Our contributions are as follows: • We introduce Next-Chapter Prediction, a new task for long-form creative writing. • We propose VR-CLI, a proxy reward formulation for reasoning

Read the complete text