Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry

Shanshan Wang Junchao Wu Fengying Ye Jingming Yao, Lidia S. Chao Derek F. Wong · 2025

Read Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry by Shanshan Wang Junchao Wu Fengying Ye Jingming Yao, Lidia S. Chao Derek F. Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01620.txt.

Introduction

Machine-made poems in modern Chinese are now hard to tell from human work. That trend disrupts the poetry world. Detectors built for other text may not transfer to poems. Shanshan Wang, with five co-authors, builds the first benchmark for this task. It pairs eight hundred poems by six professional poets with many model-made poems. Four mainstream models wrote forty-one thousand six hundred of them.

Six detectors were assessed on that data in a systematic way. Current detectors, the team finds, cannot serve as reliable tools here. The study also names the poetic traits hardest to spot as machine-made. The benchmark is a needed yardstick for a genre whose traits defeat prose-trained detectors. Brevity, imagery, and loose form all play a part. The work also prompts detectors built with poetry in mind.

Catalog record

CREATORShanshan Wang Junchao Wu Fengying Ye Jingming Yao, Lidia S. Chao Derek F. Wong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01620

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry Shanshan Wang Junchao Wu Fengying Ye Jingming Yao, Lidia S. Chao Derek F. Wong Abstract The rapid development of advanced large language models (LLMs) has made AI-generated text indistinguishable from human-written text. Previous work on detecting AI-generated text has made effective progress, but has not involved modern Chinese poetry. Due to the distinctive characteristics of modern Chinese poetry, it is difficult to identify whether a poem originated from humans or AI. The proliferation of AI-generated modern Chinese poetry has significantly disrupted the poetry ecosystem. Based on the urgency of identifying AI-generated poetry in the real Chinese world, this paper proposes a novel benchmark for detecting LLMs-generated modern Chinese poetry. We first construct a high-quality dataset, which includes both 800 poems written by six professional poets and 41,600 poems generated by four mainstream LLMs. Subsequently, we conduct systematic performance assessments of six detectors on this dataset. Experimental results demonstrate that current detectors cannot be used as reliable tools to detect modern Chinese poems generated by LLMs. The most difficult poetic features to detect are intrinsic qualities, especially style. The detection results verify the effectiveness and necessity of our proposed benchmark. Our work lays a foundation for future detection of AI-generated poetry. 1 1 1 Data and code are available at: https://github.com/NLP2CT/AIGenPoetry-Detection 1 Introduction Large language models (LLMs) have made non-negligible progress in various tasks Lan et al. 2024 ; Yan et al. 2024 ; Fang et al. 2023 ; Fang et al. 2025 ; Lyu et al. 2025 . However, the rapid development of advanced LLMs has made AI-generated text indistinguishable from human-written text Hayawi et al. 2024 ; Najjar et al. 2025 . This AI-generated content could deceive readers through novel forms of manipulation Weidinger et al. 2022 ; Buchanan et al. 2021 ; Cooke 2018 , which makes reliable detection of AI-generated text a challenge Wu et al. 2025 . This phenomenon extends even to the domain of poetry, traditionally considered the exclusive realm of human creative expression Bena and Kalita 2020 ; Chaudhuri et al. 2024 . AI-generated poetry has become increasingly prevalent in the real world. It is foreseeable that the total amount of poetry generated by AI will surpass the cumulative amount of all poetry created in human history Huo 2020 . Professional models were shown to have the potential to generate high-quality poems that are engaging to readers Linardaki 2022 ; Köbis and Mossink 2021 . However, humans have been demonstrated to exhibit limited accuracy in distinguishing between AI-generated poetry and human-written poetry Darda et al. 2023 ; Jakesch et al. 2023 . Empirical studies indicate that humans cannot distinguish poems generated by ChatGPT-3.5 and may misclassify AI-generated poems as human-written literary works Porter and Machery 2024 . With the development of literature, modern Chinese poetry has become one of the most popular literary genres in the current Chinese world. Modern Chinese poetry is a unique genre of poetry, which is obviously different from classical Chinese poetry and rhymed English poetry Wang et al. 2024 . Specifically, modern Chinese poetry is free in form and innovative in language, and is not restricted by format, sentence length, rhythm, or meter Guo 1957 ; Wang 2006b ; Skerratt 2013 ; Xu 2015 ; Awan and Khalida 2015 . Figure 1: The framework of our proposed benchmark. Can detection models accurately identify AI-generated modern Chinese poetry? The meaningful evaluation of LLMs must be grounded in cultural context Lan et al. 2025 . Previous work on detecting LLM-generated text has made effective progress Chen et al. 2025 , but these methods are not applicable to modern Chinese poetry. For instance, Uchendu et al. 2023 proposed that incoherence in text can serve as an indicator of LLM generation, but this metric fails to generalize to modern Chinese poetry because coherence and fluency are not necessary factors for modern Chinese poetry. In recent work, Wu et al. 2024a introduced GECSCORE, a detection method based on grammatical error correction scores, to distinguish between LLM-generated and human-written texts. Their core idea relies on the disparity in grammatical errors as a discriminative feature, because they found that human-written texts typically contain more grammatical errors than those produced by LLMs. This method performs well across diverse real-world text sources. However, GECSCORE is still not applicable to modern Chinese poetry because one of the most unique intrinsic features of modern poetry is its deliberate violation of grammatical conventions Deng 2007 ; Blohm et al. 2018 . Modern poets achieve maximum rhetorical tension and novel aesthetics by breaking grammatical rules Chen 2012b . Therefore, the seemingly incorrect grammar in modern poetry is actually the result of careful design, rendering grammar-based detection methods ineffective. Recently, there are authors who submit AI-generated poems to journals or publish them on online platforms without indicating that they come from LLMs. The proliferation of AI-generated modern Chinese poetry has significantly disrupted the poetry ecosystem: (1) it deceives both readers and journal editors, and (2) misleads poetry novices. Therefore, developing reliable text source identification techniques is critically urgent and practically significant for distinguishing human-written poetry from AI-generated poetry. However, no detection research has focused on modern Chinese poetry. Based on the urgency of identifying AI-generated poetry in the real Chinese world, in this paper, we propose a novel benchmark for detecting LLMs-generated modern Chinese poetry. Our goal is to determine the recognition capabilities of detectors on LLMs-generated modern Chinese poetry. We first constructed a high-quality modern Chinese poetry dataset named AIGenPoetry, which contains 800 poems written by six professional poets and 41,600 poems generated by four mainstream LLMs. This dataset focuses on different aspects of poetry, including intrinsic qualities, external structures, and emotions. Then, we conducted detection experiments on this dataset using six detectors. The experimental results demonstrate that current detectors cannot be used as reliable tools to detect modern Chinese poems generated by LLMs. Among them, the most difficult poetic features to detect are intrinsic qualities, especially style. When conducting in-domain detection experiments on poems with different features generated by different models, RoBERTa-based detector has the best comprehensive detection performance. For data with different characteristics, poems with the same style as human poems are the most difficult to detect, while poems that literally express specific emotions, especially fear, are the easiest to detect. These detection results verify the effectiveness and necessity of our proposed benchmark. Main contributions of our work are as follows: • We constructed the first modern Chinese poetry dataset that includes both high-quality poems written by professional poets and those generated by several mainstream LLMs. • We propose the first benchmark for detecting LLMs-generated modern Chinese poetry. Both in-domain and out-of-domain experimental results from six detectors verify the effectiveness and necessity of our work. • Our work reveals the vulnerabilities of existing detectors in this task, and lays a foundation for future detection of AI-generated poetry. 2 Related Work With the improvement in the quality of AI-generated poetry, some studies have made efforts to identify AI-generated poetry, but primarily focused on English poetry. Köbis and Mossink 2021 evaluated humans’ ability to discriminate English poems generated by GPT-2 Radford et al. 2019 that meet appearance criteria such as rhyme and repetition. They

Read the complete text