CreativityPrism: A Holistic Benchmark for Large Language Model Creativity

Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu, Bhiman Kumar Baghel, Anneliese Brei, Ximing Lu, Meng Jiang, Faeze Brahman, Snigdha Chaturvedi, Haw-Shiuan Chang, Daniel Khashabi, Xiang Lorraine Li · 2025

Read CreativityPrism: A Holistic Benchmark for Large Language Model Creativity by Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu, Bhiman Kumar Baghel, Anneliese Brei, Ximing Lu, Meng Jiang, Faeze Brahman, Snigdha Chaturvedi, Haw-Shiuan Chang, Daniel Khashabi, Xiang Lorraine Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20091.txt.

Introduction

Zhaoyi Joey Hou and eleven co-authors present CreativityPrism, a framework for judging how creative large language models are. Earlier checks were split across fields, leaned heavily on human raters, or used conflicting ideas of what creativity means. Their 2025 work brings those scattered checks into one shared system that can scale across many tasks and models.

CreativityPrism groups eight tasks from three areas: divergent thinking, creative writing, and logical reasoning. It grades outputs on quality, novelty, and diversity, using automatic judges checked against human ratings. In tests of seventeen leading models, larger frontier models led on writing and reasoning but not on divergent thinking. Scores in one area rarely carried over to another, which supports the need for a broad test.

Catalog record

CREATORZhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu, Bhiman Kumar Baghel, Anneliese Brei, Ximing Lu, Meng Jiang, Faeze Brahman, Snigdha Chaturvedi, Haw-Shiuan Chang, Daniel Khashabi, Xiang Lorraine Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20091

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CreativityPrism: A Holistic Benchmark for Large Language Model Creativity Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu, Bhiman Kumar Baghel, Anneliese Brei, Ximing Lu, Meng Jiang, Faeze Brahman, Snigdha Chaturvedi, Haw-Shiuan Chang, Daniel Khashabi, Xiang Lorraine Li Abstract Creativity is often seen as a hallmark of human intelligence. While large language models (LLMs) are increasingly perceived as producing creative text, there is still no holistic framework to evaluate their creativity across diverse scenarios. Existing evaluation methods remain fragmented, with dramatic variation across domains and tasks, largely due to differing definitions and measurements of creativity. Inspired by the hypothesis that creativity is not one fixed idea, we propose, CreativityPrism , an evaluation analysis framework that decomposes creativity into three dimensions: quality, novelty, and diversity. CreativityPrism incorporates nine tasks, three domains, i.e., divergent thinking, creative writing, and logical reasoning, and twenty evaluation metrics, which measure each dimension in task-specific, unique ways. We evaluate 17 state-of-the-art (SoTA) proprietary and open-sourced LLMs on CreativityPrism and analyze the performance correlations among different metrics and task domains. Our results reveal a notable gap between proprietary and open-source models. Overall, model performance tends to be highly correlated across tasks within the same domain and less so across different domains. Among evaluation dimensions, diversity and quality metrics show strong correlations—models that perform well on one often excel on the other—whereas novelty exhibits much weaker correlation with either. These findings support our hypothesis that strong performance in one creativity task or dimension does not necessarily generalize to others, underscoring the need for a holistic evaluation of LLM creativity. 1 1 1 Project website: https://joeyhou.github.io/CreativityPrism/ . 1 Introduction Creativity, the capacity to generate novel and valuable ideas or solutions [ 25 , 5 , 15 ] , is a core human cognitive ability. It appears across many domains: crafting stories with surprising plot twists [ 30 , 3 ] , producing groundbreaking scientific discoveries [ 26 , 55 ] , solving problems under constraints [ 42 , 62 ] , or even expressing humor in everyday life [ 23 , 67 ] . Its multifaceted nature has prompted extensive study in psychology and cognitive science, with efforts to capture creativity through both qualitative and quantitative approaches [ 21 , 46 , 1 , 57 ] . Recently, with the rapid rise of general-purpose LLMs, interest has grown in probing their creativity [ 65 , 17 , 8 , 42 , 3 ] . But as with human creativity, creativity spans such diverse and expansive contexts, making it difficult to define , formalize , and, above all, measure . More concretely, the evaluation of LLM creativity faces two main challenges: the difficulty of scalable, automatic evaluation due to the convoluted nature of creativity and the distinct definition of creativity across different domains. The former calls for effective automatic evaluation methods, as many existing works [ 61 , 8 ] heavily rely on human evaluation, which is expensive, inaccessible, and also time-consuming. As new LLMs are coming out nearly every day, a scalable and accessible way of evaluating LLM creativity is necessary. The latter requires a comprehensive evaluation framework that incorporates evaluation of creativity from various dimensions, as current works are scattered across different domains and thus often target narrow or singular dimensions, failing to capture shortcomings in other equally important dimensions of creativity. For example, the Divergent Association Task (DAT) [ 10 , 4 ] and the Creative Short Story Task [ 29 ] emphasize lexical diversity, yet LLMs can exploit them by generating random, incoherent words. The Creativity Index [ 40 ] compares model outputs with pre-training corpus at the n-gram level, but risks overestimating or misjudging creativity assigned to paraphrased text and models trained with private data. The Alternative Use Test (AUT) [ 17 , 48 ] solely focuses on unconventional ideas of using daily items, overlooking the pragmatics of those solutions. These are just examples from the wide range of evaluation protocols for creativity, as shown at the bottom of Figure. 1 . Such task-specific, ad hoc choices often yield inconsistent conclusions about creativity, obscuring their actual creative capacity. Together, these challenges underscore a central point: creativity is multidimensional, necessitating a holistic evaluation framework that can scale — one that integrates multiple tasks and metrics to capture quality, novelty, and diversity in a unified way. Figure 1 : Overview of CreativityPrism . We evaluated 17 LLMs across nine datasets spanning three domains: divergent thinking, creative writing, and logical reasoning. Each LLM was prompted to complete the tasks, and their outputs were evaluated using task-specific metrics. However, these metrics are diverse and difficult to interpret holistically in terms of machine creativity. To address this, we organize the metrics into three key dimensions of creativity: quality , novelty , and diversity . Creativity cannot be captured by a single measure—it must be evaluated through multiple dimensions. Task details can be found in Table 1 . To this end, we propose a holistic, scalable evaluation framework for LLM creativity evaluation, CreativityPrism , consisting of 9 tasks and 20 metrics. These tasks encompass domains such as logical reasoning (including mathematical reasoning and coding), creative writing, and divergent thinking. Due to the complexity of creativity, no single metric could represent the concept of creativity, so our framework evaluates creativity from three distinct dimensions — quality, novelty, and diversity — which are widely recognized in prior interdisciplinary literature as core dimensions of creativity [ 56 , 1 , 25 , 28 ] . We systematically categorize existing task-specific metrics along the three dimensions to facilitate a comprehensive measurement of model creativity (Figure 1 ). Quality evaluates whether LLM generations satisfy fundamental task requirements, e.g, sentence coherence and grammatical correctness. Novelty measures the originality of solutions or content by comparing their difference from existing ones. Diversity examines the variation among generated content, capturing the model’s capacity to produce distinct outputs. Our evaluation framework taxonomy utilizes these three creativity dimensions to provide a holistic and structured view for analysis. We evaluate 17 closed-sourced and open-sourced state-of-the-art (SoTA) LLMs on CreativityPrism and found a notable performance gap between proprietary and open-sourced models, especially in logical reasoning tasks, followed by creative writing tasks. In order to better understand the connections among creativity dimensions and metrics, we also conduct a detailed analysis of the correlations among models’ performance in all creativity metrics. Results have shown that the models perform similarly in metrics from the same task or the same domain. For metrics from different domains, models perform similarly in diversity and quality dimensions, while performances in novelty dimensions are much less correlated. We believe this is due to an inherent difference in how novelty is defined in different tasks and domains. We believe that CreativityPrism lays a solid foundation for measuring machine creativity and guiding the future development of creative models. 2 Related Work Human Creativity The definition of creativity has varied across different domains. In psychology, Torrance Test of Creative Thinking (TTCT) [ 1 ] considers creativity as a combination of originality, flexibility, fluency, and elaboration. In marketing, El-Murad and West 2004 , Rosengren et al. 2020 considers advertisement creativity as the combination of usefulness and originality; on top of that, Smith et al. 2007 adds flexibility, fluency,

Read the complete text