Can Artificial Intelligence Write Like Borges? An Evaluation Protocol for Spanish Microfiction
Gerardo Aleman Manzanarez, Nora de la Cruz Arana, Jorge Garcia Flores, Yobany Garcia Medina, Raul Monroy*, Nathalie Pernelle · 2025
Read Can Artificial Intelligence Write Like Borges? An Evaluation Protocol for Spanish Microfiction by Gerardo Aleman Manzanarez, Nora de la Cruz Arana, Jorge Garcia Flores, Yobany Garcia Medina, Raul Monroy*, Nathalie Pernelle (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08172.txt.
Introduction
Gerardo Aleman Manzanarez and five co-authors ask whether machine-written short fiction can stand up to real tests of literary worth. Their paper, from two thousand twenty-five, takes Spanish microfiction as its test case. The writers note that story generation by computers has been studied for decades. Yet judging the beauty and depth of the results has had far less care.
The team presents GrAImes, a scoring method built on literary theory. It grades very short stories on theme, clarity, depth of meaning, and beauty of the language. Experts and keen readers both tried the method, and their answers were used to check it. The work gives future studies a shared way to judge machine-made fiction.
Catalog record
| CREATOR | Gerardo Aleman Manzanarez, Nora de la Cruz Arana, Jorge Garcia Flores, Yobany Garcia Medina, Raul Monroy*, Nathalie Pernelle |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08172
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Artificial Intelligence Write Like Borges? An Evaluation Protocol for Spanish Microfiction Gerardo Aleman Manzanarez, Nora de la Cruz Arana, Jorge Garcia Flores, Yobany Garcia Medina, Raul Monroy*, Nathalie Pernelle Abstract Automated story writing has been a subject of study for over 60 years. Large language models can generate narratively consistent and linguistically coherent short fiction texts. Despite these advancements, rigorous assessment of such outputs for literary merit—especially concerning aesthetic qualities—has received scant attention. In this paper, we address the challenge of evaluating AI-generated microfictions and argue that this task requires consideration of literary criteria across various aspects of the text, such as thematic coherence, textual clarity, interpretive depth, and aesthetic quality. To facilitate this, we present GrAImes: an evaluation protocol grounded in literary theory, specifically drawing from a literary perspective, to offer an objective framework for assessing AI-generated microfiction. Furthermore, we report the results of our validation of the evaluation protocol, as answered by both literature experts and literary enthusiasts. This protocol will serve as a foundation for evaluating automatically generated microfictions and assessing their literary value. keywords artificial intelligence, creative writing, evaluation protocol, microfiction, GrAImes, literary reception † † firstpage: 1 † † volume: 1 † † issue: 1 † † articlenumber: 0 † † year: 2025 † † copyright-year: 2025 † † hreflink: https://doi.org/ † † titlecitation: Can Artificial Intelligence Write Like Borges? An Evaluation Protocol for Spanish Microfiction † † authornames: Gerardo Aleman Manzanarez, Jorge Garcia Flores , Nora de la Cruz Arana, Yobany Garcia Medina, Raul Monroy and Nathalie Pernelle † † authorcitation: Aleman Manzanarez, G.; de la Cruz Arana, N.; Garcia Flores, J.; Garcia Medina, Y.; Monroy, R.; Pernelle, N. † † address: 1 Tecnologico de Monterrey, Carr. Lago de Guadalupe Km.3.5, Col. Margarita M. de Juarez, Atizapan, Estado de Mexico, Mexico 2 Centre National de la Recherche Scientifique - Laboratoire d’Informatique de Paris Nord - Université Sorbonne Paris Nord, 99 av. Jean-Baptiste Clément, 93430 Villetaneuse, France; [email protected] † † corresponding: Correspondence: [email protected] † † reftitle: References \sethlcolor lightgray 1 Introduction Technological progress in Artificial Intelligence (AI) has led to systems capable of advanced reasoning ( Guo et al. 2025 ; Jaech et al. 2024 ; Team et al. 2024 ) , multimodal understanding ( Team et al. 2024 ) and creative writing ( Porter and Machery 2024 ) . Additionally, improvements in knowledge distillation ( Hinton et al. 2015 ) and reductions in inference-time computational costs ( Team et al. 2024 ; Leslie et al. 2024 ) suggest that generative AI systems are becoming more accessible and affordable, with some scientific studies suggest that AI-generated texts may surpass human-written ones in literary quality ( Clark et al. 2018 ; Jakesch et al. 2023 ) . However, the assessment of literary quality has always been a subjective matter. A review of current methods for automatically generating and evaluating automatically generated fiction ( Alhussain and Azmi 2021 ) reveals that literary criteria are seldom considered. In particular, the concept of reception ( Iser 1979 ; Ingarden 1993 ) plays a crucial role. As such study indicates, the reception of literary texts is significantly influenced by readers’ experiences and literary expertise. Consequently, an evaluation framework grounded in literary theory is essential for effectively comparing human-authored texts with those produced by generative models. Moreover, beyond a direct human-AI comparison, it becomes relevant to assess the aesthetic quality of AI-generated fiction through a literary evaluation protocol. Current evaluation methods, which mostly come from the fields of Natural Language Processing and Machine Translation, exhibit significant limitations when applied to literary texts. These methods rely heavily on quantitative metrics such as BLEU, ROUGE, and perplexity, which fail to capture nuanced aspects of literary language, like metaphor, symbolism, and stylistic creativity. These metrics prioritize surface-level similarity over deeper semantic and aesthetic qualities, leading to inadequate assessments of text richness. While AI mechanisms can generate consistent and coherent fiction, little attention has been given to producing texts with literary value. This raises the fundamental question: what makes literature literature? This paper introduces GrAImes (Grading AI and Human Microfiction Evaluation System) 1 1 1 Grading AI Microfiction Evaluation System evaluation protocol is named after Joseph E. Grimes (born 1922), an American linguist known for his work in discourse analysis, computational linguistics, and indigenous languages of America. He developed the first automatic story generator in the early 1960s ( Ryan 2017 ) , a pioneering system that used Vladimir Propp’s analysis of Russian folk stories ( Propp 2012 ) as a grammar for story generation. a novel evaluation protocol specifically designed to assess the literary value of microfictions. GrAImes incorporates literary criteria into the assessment process, aiming to evaluate the literary quality of microfictions generated by AI, as well as those created by human authors, with or without AI assistance. GrAImes reception situation is inspired by the editorial process used to accept or reject stories submitted to the publishing industry ( Ginna 2017 ) . We chose to work with microfictions (see section 2.1 ) as a model of literary narrative, both due to the limitations of language models in generating short narratives and because their brevity facilitates the evaluation process. GrAImes consists of a questionnaire with fifteen questions designed to assess literary originality, the impact of microfiction (MF) on its readers, and, to some extent, its commercial potential. The protocol’s development involved an initial validation phase, during which a group of experts (literary scholars holding a PhD and an academic position) assessed human-written microfictions. Subsequently, GrAImes was applied to AI-generated microfictions, with evaluations conducted by a community of reading and literature enthusiasts. In both experiments, it is necessary to take into consideration that human evaluators base their assessments on their imagination and prior knowledge, often assigning higher ratings to stories they find more familiar ( Peinado and Gervás 2006 ) . From a historical creativity perspective ( Boden 2004 ) , their evaluations are shaped by comparisons with other narratives they have encountered throughout their lives. With GrAImes we propose an evaluation method that aims to not only to texts likeness but also to the their aesthetic, technical, editorial and commercial quality and value. Our findings indicate that GrAImes could become a reliable framework for assessing the literary quality of human written and AI-generated microfictions. Results from the first experiment, where literature experts evaluated anonymous, human written microfictions, suggest a correlation between the author’s expertise and the microfictions’ evaluation, with good to acceptable internal consistency in the evaluator’s judgment. This was further corroborated in a second experiment involving reading enthusiasts and experts, whose evaluations slightly favored microfictions generated by ChatGPT-3.5 over those produced by a GPT-2 baseline fine-tuned language model (see section 3.2.3 ). The cumulative results of these experiments position GrAImes as a valuable tool for aiding the validation process of microfictions, demonstrating good to acceptable internal consistency in the evaluators’ judgments. However, it is important to acknowledge the limitations of the present study. The experimental dataset of our experiments is small, and some statistical methods used for validation (ICC, Cronbach’s alpha, and Kendall’s W) are sensitive to the sample size, which might introduce bias or instability in the results. Furthermore, additional research is needed to test the applicability of GrAImes to