AffectMachine-Pop: A controllable expert system for real-time pop music generation

Kat R. Agres, Adyasha Dash, Phoebe Chua, Stefan K. Ehrlich · 2025

Read AffectMachine-Pop: A controllable expert system for real-time pop music generation by Kat R. Agres, Adyasha Dash, Phoebe Chua, Stefan K. Ehrlich (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08200.txt.

Introduction

Kat Agres and co-authors build a pop music generator the user can steer by mood, in real time. Their 2025 paper presents AffectMachine-Pop, an expert system composing retro-pop tracks to hit target levels of arousal and valence, the two axes of emotional tone. Targets can be set in advance or fed live from a listener's measured state.

Unlike black-box neural generators, an expert system exposes its controls, the point for therapy and biofeedback uses. A listening study confirmed the system lands near its emotional targets: calm songs feel calm, excited songs feel excited. The authors aim the tool at interactive installations and emotion regulation support, where a soundtrack that answers your state could help steer it. The emotional claims were tested on human ears, not just model metrics.

Catalog record

CREATORKat R. Agres, Adyasha Dash, Phoebe Chua, Stefan K. Ehrlich
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08200

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AffectMachine-Pop: A controllable expert system for real-time pop music generation Kat R. Agres, Adyasha Dash, Phoebe Chua, Stefan K. Ehrlich Abstract Music is a powerful medium for influencing listeners’ emotional states, and this capacity has driven a surge of research interest in AI-based affective music generation in recent years. Many existing systems, however, are a black box which are not directly controllable, thus making these systems less flexible and adaptive to users. We present AffectMachine-Pop , an expert system capable of generating retro-pop music according to arousal and valence values, which can either be pre-determined or based on a listener’s real-time emotion states. To validate the efficacy of the system, we conducted a listening study demonstrating that AffectMachine-Pop is capable of generating affective music at target levels of arousal and valence. The system is tailored for use either as a tool for generating interactive affective music based on user input, or for incorporation into biofeedback or neurofeedback systems to assist users with emotion self-regulation. Introduction Artificial intelligence (AI)-based music generation has rapidly gained traction in recent years, driven by advances in machine learning, computational creativity, and the growing availability of music datasets. Such systems have broad applications, from enhancing artistic creativity and aiding music composition through co-creative systems, to providing interactive entertainment in gaming and VR scenarios, and even person-centered, adaptive therapeutic applications ( Ji, Luo, and Yang 2020 ; Agres et al. 2021 ; Dash and Agres 2024 ; Civit et al. 2022 ; Wang et al. 2024 ) . Advancements in generative AI, specifically large language models (LLM) and natural language processing (NLP) technology, have led to groundbreaking advancements in the field. Generative models such as AudioLM ( Borsos et al. 2023 ) , MusicLM ( Agostinelli et al. 2023 ) , and MusicGen ( Copet et al. 2024 ) can be prompted with either text descriptions or melodies to produce high-quality audio samples that mimic diverse genres, styles and sentiments. Aside from text-based prompting, previous approaches have also achieved music generation by sampling latent spaces encoding distinct musical attributes, styles, and sentiments ( Tan and Herremans 2020 ; Gupta et al. 2023 ) . Within the field of music generation, there has been significant interest in AI-based affective music generation (AI-AMG) systems, which aim to create music with specific emotional qualities ( Grekow and Dimitrova-Grekow 2021 ; Sulun, Davies, and Viana 2022 ; Cui, Sarmento, and Barthet 2024 ; Dash and Agres 2024 ) . Through their ability to induce, mediate and enhance listener’s emotional states, AI-AMG systems widen the scope of algorithmic music generation by enabling tailored emotional engagement in applications such as mental health interventions ( Elliott, Polman, and McGregor 2011 ; Stewart et al. 2019 ; Agres et al. 2021 ) , assistive technology ( Hagerer et al. 2015 ) , and physical rehabilitation ( Ashoori, Eagleman, and Jankovic 2015 ) . While generative AI has brought about important advancements in the development of AI-AMGs, specifically regarding the high quality and naturalness of the generated music, most approaches relying on generative AI (especially neural network-based approaches) suffer from several drawbacks, namely: 1) the generation of musical samples from generative AI models is typically performed at the song level and usually requires several seconds up to minutes; 2) near real-time inference for instantaneous music generation is rare; and 3) the music generation process is non-deterministic, rendering fine-grained control of the affective qualities of the music challenging, and the expected emotional impact on the listener non-trivial. Furthermore, the majority of generative AI models rely on pre-existing music for training, which has sparked controversies surrounding copyrighted training data, making approaches not relying on pre-existing music increasingly attractive. These factors render state-of-the-art neural networks-based generative AI music systems sub-optimal for real-time music generation, which is essential for interactive entertainment in gaming and VR scenarios, as well as human-in-the-loop adaptive therapeutic systems. The challenge of narrative adaptability ( Dash and Agres 2024 ) – the ability of AI-AMG systems to instantaneously generate music seamlessly adapting to a given emotional narrative or sequence, while still maintaining musical coherence – has not been explored in great depth. However, real-time adaptation of musical output, which can in turn influence the listener’s emotions, is a key area given music’s significant impact on human emotion ( Juslin and Sloboda 2013 ) . Few systems have addressed this challenge. For example, Engels, Tong, and Chan 2015 employed hierarchical Markov models to generate original music for video games, capable of mimicing the variations in musical structure often found in longer human-composed pieces. Similarly, Ehrlich et al. 2019 developed a rule-based probabilistic music generation algorithm as part of a closed-loop brain-computer interface system. The algorithm generates a stream of MIDI events, with the type and occurrence of events modulated by emotional arousal and valence parameters. Recent work by Agres, Dash, and Chua 2023 introduced an AI-AMG system called AffectMachine-Classical , capable of creating non-repetitive, continuous affective music in a classical style. This system was validated in a study with human participants, confirming the emotional qualities of the generated musical and supporting its utility for the aforementioned use cases. Building upon AffectMachine Classical, we present here AffectMachine-Pop , a novel AI-AMG system designed to generate continuous affective music in a pop style. Compared with classical music, pop music is characterized by simpler, more repetitive forms that resonate more broadly with the general public ( Chan, Qu, and Mak 2009 ) . However, generating multi-track pop music presents the technical challenge of maintaining melodic, harmonic and rhythmic coherence across several musical layers ( Ren et al. 2020 ; Zhu et al. 2018 ) . Therefore, unlike most modern pop music AI-AMGs, which predominantly rely on data-driven methods and high-performance AI architectures, e.g. ( Ren et al. 2020 ; Huang and Yang 2020 ) , our approach takes a different path. Existing pop systems often depend heavily on pre-existing music datasets and suffer from limited transparency, functioning as black-box models. In contrast, our method features a compositional framework guided by established musical rules, deliberate instrumentation, and precise timbral arrangements. These elements are manually refined by expert musicians and composers to achieve a balance of technical accuracy and artistic expressiveness. In addition, our approach features near real-time adaptability, enabling the system to seamlessly adjust its affective content based on the user’s inputs or physiological states, by traversing the valence-arousal space ( Russell 1980 ) while preserving musical and stylistic coherence. The system generates musical samples with controllable emotional qualities, i.e., music at specific levels of arousal and valence, as demonstrated by empirical evidence from the listening study presented here. Below we describe AffectMachine-Pop , our expert system for affective automatic music generation in a popular music style. We also highlight the ways in which the system compares and differs from AffectMachine-Classical. Finally, we present the methodology and results of a listening study conducted to validate the system’s ability to produce music at different target levels of arousal and valence. AffectMachine-Pop system description Here we describe the parameters and design of our system, AffectMachine-Pop , for computationally-generating affective pop music. Unlike the majority of AI-AMG systems, which have focused on generating music in a classical genre (e.g., Agres, Dash, and Chua 2023

Read the complete text