DramaBench: A Six-Dimensional Evaluation Framework for Drama Script Continuation

Shijian MA, Yunqi HUANG, Yan LIN · 2025

Read DramaBench: A Six-Dimensional Evaluation Framework for Drama Script Continuation by Shijian MA, Yunqi HUANG, Yan LIN (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19012.txt.

Introduction

Shijian Ma, Yunqi Huang, and Yan Lin created DramaBench, a test suite for AI models that continue a drama script. A good continuation must keep characters in character, move the plot, and hold the dramatic shape. Older tests, the authors say, check none of this in a full way.

DramaBench scores six separate traits, from format and logic to emotion and conflict handling. It mixes rule checks, model labeling, and statistics. The team ran eight leading models through more than a thousand scripts, and checked scores against human judges. Each trait proved to measure something distinct, giving model builders clear, separate feedback.

Catalog record

CREATORShijian MA, Yunqi HUANG, Yan LIN
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-19012

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DramaBench: A Six-Dimensional Evaluation Framework for Drama Script Continuation Shijian MA, Yunqi HUANG, Yan LIN Abstract Drama script continuation requires models to maintain character consistency, advance plot coherently, and preserve dramatic structure—capabilities that existing benchmarks fail to evaluate comprehensively. We present DramaBench , the first large-scale benchmark for evaluating drama script continuation across six independent dimensions: Format Standards, Narrative Efficiency, Character Consistency, Emotional Depth, Logic Consistency, and Conflict Handling. Our framework combines rule-based analysis with LLM-based labeling and statistical metrics, ensuring objective and reproducible evaluation. We conduct comprehensive evaluation of 8 state-of-the-art language models on 1,103 scripts (8,824 evaluations total), with rigorous statistical significance testing (252 pairwise comparisons, 65.9% significant) and human validation (188 scripts, substantial agreement on 3/5 dimensions). Our ablation studies confirm all six dimensions capture independent quality aspects (mean | r | = 0.020 |r|=0.020 ). DramaBench provides actionable, dimension-specific feedback for model improvement and establishes a rigorous standard for creative writing evaluation. Figure 1: Overview of the DramaBench evaluation framework. The pipeline consists of three components: (1) Task input with script context and model continuation, (2) Six independent evaluation dimensions (Format, Narrative, Character, Emotion, Logic, Conflict), and (3) Structured LLM labeling framework that extracts categorical labels (not direct scores) which are then aggregated into objective metrics. This approach ensures reproducibility and provides actionable feedback for model improvement. 1 Introduction Drama scripts are a unique form of creative writing that combine narrative storytelling with specific structural constraints. Unlike general prose or dialogue generation, drama script continuation demands that language models simultaneously handle multiple quality dimensions: maintaining established character personalities, advancing plot through meaningful events, preserving logical consistency with prior context, managing emotional arcs, and escalating dramatic conflicts—all while adhering to screenplay formatting standards. Despite the growing capabilities of large language models in creative writing tasks, existing benchmarks for evaluating script generation focus primarily on full-script generation from scratch 16 ; 30 or lack quantitative, multi-dimensional evaluation frameworks. Story generation benchmarks like ROCStories 18 evaluate story coherence through cloze tests but do not capture the screenplay-specific requirements of format compliance, character voice distinctiveness, or dramatic structure. General NLG evaluation frameworks like UniEval 29 provide multi-dimensional assessment but are not specialized for the unique challenges of drama script continuation. This gap creates three critical challenges for evaluating drama script continuation systems: (1) Multi-dimensional quality : Script quality cannot be captured by a single aggregate score—a script may have perfect format but poor characterization, or vice versa; (2) Subjective evaluation : Traditional human evaluation of creative writing is expensive, slow, and suffers from low inter-annotator agreement; (3) Lack of actionable feedback : Aggregate quality scores fail to identify specific weaknesses for model improvement. We address these challenges with DramaBench, a comprehensive benchmark that evaluates drama script continuation through six independent dimensions using a novel LLM Labeling + Statistical Analysis methodology (Figure 1 ). Unlike LLM-as-a-Judge approaches that ask models to directly score quality 28 , our framework uses LLMs as structured data annotators that extract categorical labels (e.g., “driver beat” vs “static beat”), which are then converted to objective metrics through statistical formulas. This approach ensures reproducibility, interpretability, and enables the extracted labels to serve as training data for model improvement. Our contributions are: 1. DramaBench benchmark : The first large-scale benchmark for drama script continuation, containing 1,103 professionally-structured scripts with scene-boundary-aware context-continuation splits (8,824 model-script evaluations across 8 SOTA models). 2. Six-dimensional evaluation framework : A hybrid evaluation system combining rule-based analysis (Format Standards) with LLM-labeled dimensions (Narrative Efficiency, Character Consistency, Emotional Depth, Logic Consistency, Conflict Handling), each with precisely defined annotation units and statistical metrics. 3. Rigorous validation : Comprehensive statistical significance testing (252 Mann-Whitney U tests with FDR correction, 65.9% significant comparisons), ablation studies confirming dimension independence (mean correlation |r|=0.014 ), and human validation on 188 scripts showing substantial agreement on 3/5 dimensions ( \kappa=0.42 – 0.53 ). 4. Systematic analysis : In-depth error taxonomy classifying 10,850 errors, 24 case studies demonstrating model strengths/weaknesses, and model-specific error profiles revealing actionable improvement opportunities. 5. Public release : All evaluation scripts, metrics, and analysis code will be released to enable reproducible drama generation research. Our evaluation reveals that no single model excels across all dimensions: GPT-5.2 leads in overall robustness (narrative efficiency, character consistency, logic), Qwen3-Max specializes in emotional depth, and Gemini-3-Pro excels at conflict management. All models achieve near-perfect format compliance, but show significant variance in logic consistency (2–5% error rates). These findings demonstrate that multi-dimensional evaluation is essential for understanding model capabilities and guiding targeted improvements. 2 Related Work 2.1 Story and Drama Generation Early story generation benchmarks focused on commonsense reasoning and narrative coherence. ROCStories 18 introduced a 100K-story corpus with the Story Cloze Test, evaluating models’ ability to select the correct ending from alternatives. While foundational, this approach evaluates understanding rather than generation, and does not address screenplay-specific requirements like format compliance or character voice. Recent work has explored drama and screenplay generation systems. Dramatron 16 uses hierarchical LLM prompting to generate complete scripts with scene breakdowns, character descriptions, and dialogue, validated through a 15-expert user study. IBSEN 30 proposes a director-actor agent framework where a director agent plans plot points and actor agents generate character-specific dialogue. However, these are generation systems rather than benchmarks —they lack standardized test sets, quantitative metrics, or large-scale model comparisons. 2.2 Multi-Dimensional NLG Evaluation UniEval 29 frames NLG evaluation as boolean question answering, training a unified model to assess coherence, consistency, fluency, and relevance across multiple tasks. While general-purpose, it does not capture screenplay-specific dimensions like dramatic structure or character voice distinctiveness. LitBench 6 evaluates creative writing quality but focuses on prose rather than screenplay format. Recent surveys on LLM-as-a-Judge 10 ; 11 highlight reliability concerns when LLMs directly score outputs, noting issues with position bias, verbosity bias, and inconsistent criteria application. We address these concerns by using LLMs for structured labeling rather than direct scoring . 2.3 Evaluation Benchmarks MT-Bench 28 evaluates conversational AI through multi-turn dialogues judged by GPT-4, demonstrating strong correlation with human preferences. However, its focus on instruction-following and factual correctness differs from creative writing evaluation. HelloBench 20 assesses long-context generation capabilities but lacks domain-specific metrics for dramatic structure. Positioning: DramaBench is the first benchmark to combine (1) large-scale quantitative evaluation (8,824 samples), (2) drama-specific multi-dimensional metrics, (3) statistical significance validation, and (4) human-LLM agreement analysis for screenplay continuation. 3 Dataset 3.1 Data Collection and Processing DramaBench is constructed from a curated subset of 1,103 English drama scripts sourced from the english_short_drama_scripts_dataset , which contains professionally-written short-form drama scripts following Fountain screenplay format. We selected high-quality scripts through a filtering pipeline that removed duplicates, truncated scripts, and format-corrupted samples. 3.2 Scene-Boundary-Aware Splitting To create continuation tasks, we split each script into context and continuation segments using a scene-boundary-aware algorithm: 1. Default split : Identify the 50% midpoint (by line count) 2. Boundary optimization : Search within a ±20% window for scene heading markers ( INT. or EXT. ) 3. Final split : Use scene boundary if found (69.5% of cases), otherwise use midpoint This approach ensures that continuations begin at natural narrative breakpoints, improving the realism of the generation task. 3.3 Dataset Statistics Table 1 summarizes DramaBench’s key statistics. With an average context of 381

Read the complete text