Evaluation Framework for AI Creativity: A Case Study Based on Story Generation

Pharath Sathya Yin Jou Huang Fei Cheng Graduate School of Informatics, Kyoto University, Kyoto, Japan {pharath, huang, feicheng}@nlp.ist.i.kyoto-u.ac.jp · 2026

Read Evaluation Framework for AI Creativity: A Case Study Based on Story Generation by Pharath Sathya Yin Jou Huang Fei Cheng Graduate School of Informatics, Kyoto University, Kyoto, Japan {pharath, huang, feicheng}@nlp.ist.i.kyoto-u.ac.jp (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03698.txt.

Introduction

Pharath Sathya, Yin Jou Huang, and Fei Cheng propose a structured way to judge creativity in AI-generated stories. Standard metrics compare text against references, which misses the subjective heart of creativity. Their scheme scores four parts, Novelty, Value, Adherence, and Resonance, broken into eleven sub-parts, and tests them through controlled story generation.

Using a prompting method they call Spike Prompting, they got judgments from one hundred fifteen readers in a crowdsourced study. Creativity turned out to be judged in stages rather than as a simple sum, with varied dimensions mattering at first read and on reflection. Reflective judging also changed ratings and agreement among readers. The scheme exposes creative traits that reference scores flatten, helping writers and tool builders judge made-up fiction more honestly.

Catalog record

CREATORPharath Sathya Yin Jou Huang Fei Cheng Graduate School of Informatics, Kyoto University, Kyoto, Japan {pharath, huang, feicheng}@nlp.ist.i.kyoto-u.ac.jp
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03698

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluation Framework for AI Creativity: A Case Study Based on Story Generation Pharath Sathya Yin Jou Huang Fei Cheng Graduate School of Informatics, Kyoto University, Kyoto, Japan {pharath, huang, feicheng}@nlp.ist.i.kyoto-u.ac.jp Abstract Evaluating creative text generation remains a challenge because existing reference-based metrics fail to capture the subjective nature of creativity. We propose a structured evaluation framework for AI story generation comprising four components—Novelty, Value, Adherence, and Resonance—and eleven sub-components. Using controlled story generation via “Spike Prompting” and a crowdsourced study of 115 readers, we examine how different creative components shape both immediate and reflective human creativity judgments. Our findings show that creativity is evaluated hierarchically rather than cumulatively, with different dimensions becoming salient at different stages of judgment, and that reflective evaluation substantially alters both ratings and inter-rater agreement. Together, these results support the effectiveness of our framework in revealing dimensions of creativity that are obscured by reference-based evaluation. 1 Introduction The evaluation of Open-Ended Creative Generation, particularly in story writing, presents a distinct challenge compared to analytical tasks. While code generation or mathematical reasoning can be verified against objective ground truths, creative writing lacks a single correct solution. As 1 posits, creativity is not an intrinsic property measurable by a ruler, but a judgment dependent on the subjective agreement of appropriate observers. Despite this subjectivity, current evaluation paradigms often seek to objectify the process. Recent reference-based approaches (e.g., 3 ) evaluate AI stories by measuring their proximity to human “gold standards” like The New Yorker. However, this presents a methodological limitation: since creativity often requires diverging from established norms, metrics that reward similarity to a reference inadvertently penalize original outputs that drift “too far” from the baseline. To capture this necessary divergence from established definitions, we developed a domain-specific framework for AI story generation refined from the theoretical model of 2 . We validated this framework through a crowdsourcing study ( N=115 ) utilizing “Spike Prompting”, a controlled generation strategy designed to isolate distinct creative dimensions. Our analysis yields three primary insights. First, creativity judgments follow an ordered evaluation process rather than a purely additive one : users rely on emotional response in initial impressions, but shift toward originality and constraint satisfaction when asked to reflect. Second, analytical reflection alters both ratings and rater agreement , increasing appreciation for technically constrained content while reducing ratings for emotionally driven stories; mid-range scores often reflect disagreement rather than consensus. Third, Creativity and Enjoyment are driven by different criteria , such that features that improve perceived creativity do not necessarily increase enjoyment, revealing a systematic misalignment between creativity-oriented optimization and user preference. 2 Methodology 2.1 Creativity Components We adopted the domain-specific theoretical framework of 2 , refining four orthogonal dimensions operationalized for Generative AI in story writing, decomposing them into 11 granular sub-components: Adherence (Topic/Tone Fidelity), Novelty (Vocabulary Freshness, Plot Uniqueness, Surprise), Technical Value (Logical Coherence, Stylistic Quality), and Resonance (Emotional Impact, Thought-Provocation, Empathy). 2.2 Story Generation We constructed a controlled corpus of 12 narratives using Gemini 3.0 Pro (Preview) with default settings (350–450 words; Min: 366, Max: 470). We employed “Spike Prompting”: System Instructions maximized one orthogonal dimension while sacrificing others, mapping each spike to a specific narrative tone— Surreal (Spike Novelty: dream logic, bizarre metaphors), Clinical (Spike Adherence: objective descriptions, strict logic), Melancholic (Spike Resonance: emotional depth, memory), and Witty (Spike Technical Value: linguistic competence, sarcasm). We applied these four conditions across three topics ( AI Shutdown, The Heist, The Midnight Store ) to ensure findings were genre-agnostic (for full details, see Appendix B and D ). 2.3 Readability Control We calculated Flesch-Kincaid Grade Levels for all stimuli. The corpus averaged Grade 7.7 (approx. 12–13 years old). While Clinical Tone stories were more complex (Grade 9.0) than Melancholic (Grade 6.7), all remained within general literacy limits, confirming that rating differences reflect stylistic preference rather than incomprehensibility (see Appendix B for full readability analysis). 2.4 Human Evaluation Procedure We recruited N=115 participants via Prolific using a between-subjects design (each participant evaluated one story). The evaluation consisted of three stages: (1) Initial Ratings (immediate, holistic ratings for Overall Creativity and Personal Enjoyment, 1–7 Likert), (2) Component Analysis (evaluation of the 11 specific sub-components), and (3) Reflective Rating (final Reflective Creativity score after analytical processing). See Appendix A for full survey design, ethical procedures, and questionnaire details. 3 Results 3.1 Measurement Validation To validate that our constructs measure distinct creative dimensions rather than a single “general quality” factor, we conducted reliability analysis and manipulation checks (N=115). All statistical analyses are detailed in Appendix C . Internal Consistency. Cronbach’s \alpha confirmed that sub-components reliably aggregate into their theoretical constructs: Resonance ( \alpha=0.82 , 95% CI [0.76, 0.87]), Technical Value ( \alpha=0.80 , CI [0.73, 0.86]), Novelty ( \alpha=0.76 , CI [0.67, 0.83]), and Adherence ( \alpha=0.69 , CI [0.56, 0.79]). All exceeded the 0.70 threshold for acceptable reliability except Adherence, which remained within acceptable range for exploratory research ( 4 ) . Construct Validity. Manipulation checks confirm that Spike Prompting successfully isolated distinct creative dimensions (Appendix B.4). Each tone reliably maximized its intended construct while suppressing others: Clinical tone emphasized Adherence at the expense of Resonance, Surreal tone maximized Novelty, and Melancholic tone maximized Resonance. Witty tone showed more balanced performance across dimensions, suggesting that stylistic sophistication can engage multiple constructs simultaneously. Together, these patterns indicate that the framework captures separable creativity components rather than a single undifferentiated quality signal. Tone Adherence Resonance Novelty Value Clinical 5.95 * 3.54 * 4.46 4.34 Melancholic 5.88 5.03 * 4.59 5.15 Surreal 5.10 4.16 5.29 * 4.34 Witty 5.48 4.54 4.70 4.93 *Indicates targeted construct for each condition. Table 1: Construct means by tone condition. Diagonal entries indicate targeted constructs. Figure 1: Reversal in Evaluation Order. Feature importance comparison between Initial (Blue) and Reflective (Red) creativity. 3.2 Which Creativity Dimensions Matter First? A Hierarchical Analysis While linear regression identifies feature importance, it assumes a cumulative relationship. To identify hierarchical dependencies—i.e., which dimensions are evaluated first and which only matter conditionally—we trained two Decision Tree Regressors ( R^{2}>0.72 ) for Initial and Reflective Creativity. A Reversal in Evaluation Order. The observed shift reflects a change in evaluation order rather than simple reweighting (Figure 1 ). Initial judgments are dominated by Resonance, with Novelty contributing little in the absence of emotional engagement. Upon reflection, this order reverses: Novelty becomes the primary criterion, shifting the evaluative focus from emotional response to originality. The Re-emergence of Adherence. Adherence plays no role in initial impressions but becomes relevant during reflection, indicating a delayed constraint check. Users initially prioritize emotional response, but later penalize instruction violations once they engage in analytical evaluation. 3.3 The Instability of Agreement: Reflection Impact & Overall User Agreement The Reflection Gap. As shown in Figure 2 (Left), we observed a divergence in how analytical thinking alters judgment. Clinical Tone (High adherence) ratings increased upon reflection (+0.30), as analytical processing allows appreciation of technical difficulty. Conversely, creative tones decreased (Melancholic -0.21 ), as analytical processing strips away emotional magic. We then analyzed rating stability to test if a single “gold standard” exists for creativity. The Agreement U-Curve (Polarization). Mean ratings and standard deviation exhibit a strong negative correlation for both Initial ( r=-0.71,p=0.01 ) and Reflective ( r=-0.74,p=0.006 )

Read the complete text