AImoclips: A Benchmark for Evaluating Emotion Conveyance in Text-to-Music Generation
Gyehun Go, Satbyul Han, Ahyeon Choi, Eunjin Choi, Juhan Nam, Jeong Mi Park · 2025
Read AImoclips: A Benchmark for Evaluating Emotion Conveyance in Text-to-Music Generation by Gyehun Go, Satbyul Han, Ahyeon Choi, Eunjin Choi, Juhan Nam, Jeong Mi Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00813.txt.
Introduction
Gyehun Go leads six authors in total, with Satbyul Han, Ahyeon Choi, Eunjin Choi, Juhan Nam, and Jeong Mi Park, in introducing AImoclips, a benchmark for emotion conveyance in text-to-music generation. Systems now create expressive music from natural language prompts, but how faithfully they convey intended emotions has been little studied.
The team selected twelve emotion intents across four quadrants of the valence-arousal space, which maps feelings by pleasantness and intensity, and used six leading systems to generate over one thousand clips. One hundred eleven participants rated perceived valence and arousal on a nine-point scale. Commercial systems skew more pleasant than intended, open-source systems the opposite, high-arousal emotions convey best, and all systems bias toward neutrality.
Catalog record
| CREATOR | Gyehun Go, Satbyul Han, Ahyeon Choi, Eunjin Choi, Juhan Nam, Jeong Mi Park |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-00813
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AImoclips: A Benchmark for Evaluating Emotion Conveyance in Text-to-Music Generation Gyehun Go, Satbyul Han, Ahyeon Choi, Eunjin Choi, Juhan Nam, Jeong Mi Park Abstract Recent advances in text-to-music (TTM) generation have enabled controllable and expressive music creation using natural language prompts. However, the emotional fidelity of TTM systems remains largely underexplored compared to human preference or text alignment. In this study, we introduce AImoclips, a benchmark for evaluating how well TTM systems convey intended emotions to human listeners, covering both open-source and commercial models. We selected 12 emotion intents spanning four quadrants of the valence-arousal space, and used six state-of-the-art TTM systems to generate over 1,000 music clips. A total of 111 participants rated the perceived valence and arousal of each clip on a 9-point Likert scale. Our results show that commercial systems tend to produce music perceived as more pleasant than intended, while open-source systems tend to perform the opposite. Emotions are more accurately conveyed under high-arousal conditions across all models. Additionally, all systems exhibit a bias toward emotional neutrality, highlighting a key limitation in affective controllability. This benchmark offers valuable insights into model-specific emotion rendering characteristics and supports future development of emotionally aligned TTM systems. [orcid=0009-0006-2840-9643, [email protected] ] [orcid=0009-0000-8288-5671, [email protected] ] [orcid=0000-0001-8304-599X, [email protected] ] [orcid=0009-0002-1930-6141, [email protected] ] [orcid=0000-0003-2664-2119, [email protected] ] [orcid=0000-0001-7172-686X, [email protected] ] *1 keywords Text-to-Music Generation ,Emotion Conveyance ,Human Evaluation ,Valence–Arousal Model † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) † † venue: HCMIR25: 3rd Workshop on Human-Centric Music Information Research, September 20th, 2025, Daejeon, Korea † † address: Korea Advanced Institute of Science and Technology (KAIST) † † address: Seoul National University (SNU) † † corresp: Corresponding author. 1 Introduction Recent advances in generative AI have transformed creative domains, including music. AI music generation has evolved from unconditional van den Oord et al. 2016 ; Huang et al. 2018 ; Huang and Yang 2020 to conditional generation Roberts et al. 2018 ; Payne 2019 ; Dhariwal et al. 2020 ; Agostinelli et al. 2023 for better human controllability, with text-to-music (TTM) systems becoming prominent due to their familiar input format, user-friendliness, and expressive potential. Studies on the evaluation of TTM systems have also followed recent advances Grötschla et al. 2025 ; liu 2025 ; Huang et al. 2025 , focusing on human preferences and text prompt alignment. However, emotional aspects are still often overlooked in evaluations of TTM systems. Although several studies have addressed emotion-conditioned music generation, these have mainly focused on symbolic music Hung et al. 2021 ; Ferreira and Whitehead 2019 ; Grekow and Dimitrova-Grekow 2021 ; Sulun et al. 2022 ; Huang et al. 2024 , and the question of how well TTM systems convey intended emotions to human listeners remains largely unexamined. Neglecting emotion evaluation can lead to mismatches between user expectations and generated music, limiting the use of TTM systems in real-world applications such as affective music recommendations or adaptive soundtracks. Thus, assessing the effectiveness of emotional communication is crucial for developing user-centered AI music systems. In light of this background, we develop AImoclips, a benchmark that consists of AI-generated music clips for evaluating emotion conveyance of TTM systems. We generated 991 clips with six state-of-the-art models, and collected valence–arousal ratings from 111 human participants. We open-source the dataset, annotated with valence and arousal scores averaged across multiple raters. The complete dataset is open-source and available at https://github.com/HunRotation/HunRotation.github.io , with audio samples provided at https://hunrotation.github.io/projects/aimoclips.html . We also analyze the results to understand the characteristics of TTM systems regarding emotion conveyance, focusing on three main questions: (1) How differently do TTM systems convey intended emotions to humans? (2) What are the valence–arousal features of well- and poorly conveyed emotions? (3) How emotionally diverse are the clips generated by TTM systems? 2 Related Work Music emotion analysis has led to the development of several publicly available datasets with emotion annotations. These include categorical Hung et al. 2021 ; Turnbull et al. 2007 ; Wang et al. 2014 ; Eunjin Choi 2022 and dimensional approaches Ferreira and Whitehead 2019 ; Soleymani et al. 2013 ; Aljanaki et al. 2017 ; Speck et al. 2011 , often using valence–arousal models. Recent benchmarks such as GlobalMood Lee et al. 2025 address the cross-cultural differences in music emotion recognition. In parallel with these developments, evaluating emotional fidelity in AI music generation has gained importance. While many studies rely on automatic metrics due to the cost of human evaluation, recent works have begun incorporating human preferences. However, most research in this area still focuses on assessing automatic metrics Grötschla et al. 2025 or improving them to better align with human judgments liu 2025 ; Huang et al. 2025 . Although relatively few studies directly address emotional conveyance, there have been efforts to analyze emotion conveyance in AI-generated music. The work most similar to ours is that of Gao et al. Gao et al. 2024 , who evaluated emotion conveyance accuracy in the valence–arousal plane using discrete emotion labels. In contrast, our study uses continuous valence–arousal ratings, involves significantly more clips and subjects, and includes a broader set of systems—both open-source and commercial—providing a more comprehensive evaluation of current TTM systems. 3 Dataset Construction 3.1 Emotion Intents First, we selected 12 emotion words to represent emotional intents when generating music clips. We used two sets of words as candidates. The first set consists of 28 sample words that Russell Russell 1980 used to represent the domain of affect. The second set comprises adjectives that do not lie exactly on the valence or arousal axis, from the 12-Point Affect Circumplex Model (12-PAC) Yik et al. 2011 . From the candidate words, we selected those included in the Warriner et al.’s extension Warriner et al. 2013 of Affective Norms for English Words (ANEW) Bradley and Lang 1999 and filtered out words with valence or arousal scores between 4.0 and 6.0 on a 1.0–9.0 scale. This step was intended to exclude words with intermediate valence or arousal values, thereby clearly separating the groups of emotion intents corresponding to each quadrant of the valence-arousal plane. Finally, from the remaining words, we selected three for each quadrant, using 5.0 (both for valence and arousal) as the boundary. The selected words are listed in Table 1 . Table 1 : Words chosen for emotion intent of TTM generation Quadrant Valence Arousal Words Q1 High High happy, excited, energetic Q2 Low High angry, anxious, scared Q3 Low Low sad, gloomy, dull Q4 High Low relaxed, calm, tranquil 3.2 TTM System Selection and Clip Generation Next, we selected six TTM systems, then generated 14 clips for each emotion intent. There are 12 emotion intents, resulting in 168 clips created by each system, forming a dataset of 1,008 AI-generated music clips. All clips were 10 seconds long. We utilized four open-source models—AudioLDM 2 Liu et al. 2024 , MusicGen Copet et al. 2023 , Mustango Melechovsky et al. 2023 , and Stable Audio Open Evans et al. 2025 —and two commercial models, Suno v4.5 Suno 2024 and Udio v1.5 Allegro Udio 2024 . AudioLDM 2 and Stable Audio Open are diffusion-based generative audio models Ho