PoeTone: A Framework for Constrained Generation of Structured Chinese Songci with LLMs
Zhan Qu, Shuzhou Yuan, Michael Färber · 2025
Read PoeTone: A Framework for Constrained Generation of Structured Chinese Songci with LLMs by Zhan Qu, Shuzhou Yuan, Michael Färber (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02515.txt.
Introduction
Zhan Qu, Shuzhou Yuan, and Michael Farber study whether language models can write a strict classical Chinese poetry form in a 2025 paper. The form, Songci, follows templates that fix line lengths, tone patterns, and rhymes. It is a hard test of whether models can obey tight formal rules while still writing well.
The team built a four-part grading frame: a formal conformity score, automated quality judging, human review, and probing tasks. They ran eighteen models through it under five prompting styles, from bare requests to worked examples and step-by-step reasoning. A generate-and-critic setup then used the grader to pick the best drafts and fine-tune small models, lifting conformity by up to about six percent. The paper maps where models stand on this culturally prized form.
Catalog record
| CREATOR | Zhan Qu, Shuzhou Yuan, Michael Färber |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-02515
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PoeTone: A Framework for Constrained Generation of Structured Chinese Songci with LLMs Zhan Qu, Shuzhou Yuan, Michael Färber Abstract This paper presents a systematic investigation into the constrained generation capabilities of large language models (LLMs) in producing Songci , a classical Chinese poetry form characterized by strict structural, tonal, and rhyme constraints defined by Cipai templates. We first develop a comprehensive, multi-faceted evaluation framework that includes: (i) a formal conformity score, (ii) automated quality assessment using LLMs, (iii) human evaluation, and (iv) classification-based probing tasks. Using this framework, we evaluate the generative performance of 18 LLMs , including 3 proprietary models and 15 open-source models across 4 families, under five prompting strategies : zero-shot, one-shot, completion-based, instruction-tuned, and chain-of-thought. Finally, we propose a Generate-Critic architecture in which the evaluation framework functions as an automated critic. Leveraging the critic’s feedback as a reward signal, we fine-tune 3 lightweight open-source LLMs via supervised fine-tuning (SFT), resulting in improvements of up to 5.88% in formal conformity. Our findings offer new insights into the generative strengths and limitations of LLMs in producing culturally significant and formally constrained literary texts. Introduction Recent advancements in large language models (LLMs) have demonstrated remarkable capabilities in generating fluent, coherent, and contextually appropriate text across diverse domains ( Brown et al. 2020 ; Touvron et al. 2023 ) . From drafting emails and summarizing documents to composing fiction and poetry, models such as GPT-4o , Gemini 2.5 Pro , and DeepSeek-R1 have significantly expanded the creative potential of AI systems ( OpenAI et al. 2024 ; Comanici et al. 2025 ; DeepSeek-AI 2025 ) . Yet, tasks requiring both expressive fluency and strict adherence to formal rules, such as generating classical poetry, remain a significant challenge ( Yu et al. 2024 ) . Figure 1: The multi-layered challenge of generating high-quality Songci , highlighting the formal constraints of the Cipai as the primary bottleneck. This paper focuses on Songci , a prominent form of Chinese lyric poetry from the Song dynasty (960–1279 CE). Unlike Tang poetry, Songci follows fixed Cipai , predefined tune patterns that govern stanza count, line count, line length, tonal flow (píng and zè), rhyme type, and rhyme positions ( Ge 2009 ) . Effective Songci must not only conform to these structural constraints but also express coherent and emotionally resonant themes, typically involving longing, solitude, patriotism, or nature. This blend of metrical rigidity and artistic depth makes Songci a compelling testbed for constrained text generation. A common failure example in generating Songci is shown in Figure 1 . While prior work has applied deep learning to classical Chinese poetry, most focus on stylistic imitation or surface-level coherence ( Liao et al. 2019 ; Tang et al. 2025 ) . Whether modern LLMs can generate Songci that satisfy its structural and tonal requirements remains largely unexplored ( Song 2022 ) . Moreover, existing benchmarks lack fine-grained tools for evaluating formal correctness, and few approaches aim to improve generation under these constraints. To bridge these gaps, we introduce PoeTone , a systematic investigation into the capacity of LLMs to generate Songci while adhering to its strict metrical constraints. Our complete research pipeline is detailed in Figure 2 : First, we develop a comprehensive evaluation framework that includes: (i) a metadata resource detailing the structure and tonal patterns of 20 widely used Cipai , (ii) a curated corpus of canonical Songci poems organized by theme and Cipai , and (iii) a multi-dimensional evaluation protocol combining formal conformity scores, automated quality assessment, human ratings, and classification-based probing. Second, we benchmark 18 state-of-the-art LLMs, including 3 proprietary models ( GPT-4o , Gemini 2.5 Pro , ERNIE 4.5 Turbo ) and 15 open-source models from 4 families ( LLaMA , Mistral , Qwen , DeepSeek ), across five prompting strategies: zero-shot, one-shot, completion, instruction, and chain-of-thought. Third, we propose a Generate-Critic architecture that uses rule-based conformity scoring to guide fine-tuning. Applied to 3 open-source LLMs via best-of- N rejection sampling and LoRA-based supervised fine-tuning (SFT), this method improves adherence to structural and tonal constraints by up to 5.88% . In summary, our key contributions are: • A structured evaluation framework for Songci generation, including a metadata resource, a canonical corpus, and a multi-faceted evaluation framework. • A benchmark of 18 LLMs under five prompting strategies, revealing model limitations in conforming to formal poetic constraints. • A Generate-Critic method that improves constrained generation through automated rule-based feedback. Figure 2: An overview of our research pipeline, from framework development and benchmarking to model enhancement. Related Word Automatic generation of Chinese poetry has evolved from statistical and rule-based methods to neural networks ( Wang et al. 2016 ; Chen and Cao 2024 ) . Initial approaches used RNNs and LSTMs ( Zhang and Lapata 2014 ; Wang et al. 2016 ) , later enhanced by attention mechanisms to improve coherence and context modeling ( Yi, Li, and Sun 2017 ) . A major research focus has been controlling poetic form and content. To handle strict structural and rhyming constraints, methods include dual-encoder models conditioned on rhythmic patterns ( Luo et al. 2021 ) and form-aware generation guided by stressed weighting ( Hu and Sun 2020 ) . Thematic coherence has been addressed through planning-based structures ( Wang et al. 2016 ) , working memory models ( Yi et al. 2018 ) , salient clue guidance ( Yi, Li, and Sun 2018 ; Gao et al. 2021 ) , adversarial training for title consistency ( Li et al. 2018 ) , and graph neural networks for topic modeling ( Yan et al. 2023 ) . Sentiment and style control have been explored via multi-tag classification and conditional variational autoencoders ( Shao et al. 2021 ; Chen et al. 2019 ; Song et al. 2025 ) . The rise of large language models (LLMs) has transformed Chinese poetry generation. An early study showed that a simple Generative Pre-trained Transformer (GPT) could produce high-quality classical poetry without extensive feature engineering ( Liao et al. 2019 ) . In further studies, researchers have fine-tuned pre-trained models on specialized poetry corpora, finding that smaller, domain-specific models can sometimes outperform larger, general-purpose LLMs like GPT-4o on poetic tasks ( Tang et al. 2025 ; Wang, Guan, and Liu 2022 ) . Innovations like CharPoet introduce token-free, character-level control to improve format accuracy ( Yu et al. 2024 ) , reflecting a trend to harness LLM capabilities while imposing classical poetic constraints. Beyond text generation, interactive systems such as Jiuge ( Zhipeng et al. 2019 ) and Yu Sheng ( Ma, Zhan, and Wong 2023 ) incorporate human-in-the-loop frameworks for collaborative poem refinement. SongSong ( Hu et al. 2025 ) generates music from Songci lyrics, addressing musical restoration, but leaving high-quality, metrically-correct Songci generation as an open challenge. Our work focuses on this core task, leveraging LLMs to generate formal Songci text. Songci Generation and Evaluation Framework Evaluating the ability of LLMs to generate valid Songci requires two key resources that, to our knowledge, are unavailable in structured, machine-readable form. First, a formal constraint specification for each Cipai is necessary for automated verification. Second, a high-quality corpus of canonical Songci , thematically categorized and representative of diverse Cipai , is essential