Yin-Yang: Developing Motifs With Long-Term Structure And Controllability
Keshav Bhandari, Geraint A. Wiggins, Simon Colton · 2025
Read Yin-Yang: Developing Motifs With Long-Term Structure And Controllability by Keshav Bhandari, Geraint A. Wiggins, Simon Colton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17759.txt.
Introduction
Keshav Bhandari, Geraint Wiggins, and Simon Colton address a weak spot in computer-made music: keeping a motif developing in a planned way across a whole piece. Transformer models write note by note, so once they drift from the motif they cannot correct themselves. Their framework, Yin-Yang, splits the job among three models. One generates phrases, one refines them, and one selects among the results. The refiner is trained by corrupting good phrases and learning to repair them, so at generation time it can turn a stray phrase back into a variation of the motif.
The authors also add a score that measures how smoothly the motif shows itself through the piece. In tests, Yin-Yang beat leading transformer models while giving the user more control over the outcome. Its choices stay partly readable, which helps musical analysis. Code and a demo page are released with the paper.
Catalog record
| CREATOR | Keshav Bhandari, Geraint A. Wiggins, Simon Colton |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-17759
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Yin-Yang: Developing Motifs With Long-Term Structure And Controllability Keshav Bhandari, Geraint A. Wiggins, Simon Colton Abstract Transformer models have made great strides in generating symbolically represented music with local coherence. However, controlling the development of motifs in a structured way with global form remains an open research area. One of the reasons for this challenge is due to the note-by-note autoregressive generation of such models, which lack the ability to correct themselves after deviations from the motif. In addition, their structural performance on datasets with shorter durations has not been studied in the literature. In this study, we propose Yin-Yang, a framework consisting of a phrase generator, phrase refiner, and phrase selector models for the development of motifs into melodies with long-term structure and controllability. The phrase refiner is trained on a novel corruption-refinement strategy which allows it to produce melodic and rhythmic variations of an original motif at generation time, thereby rectifying deviations of the phrase generator. We also introduce a new objective evaluation metric for quantifying how smoothly the motif manifests itself within the piece. Evaluation results show that our model achieves better performance compared to state-of-the-art transformer models while having the advantage of being controllable and making the generated musical structure semi-interpretable, paving the way for musical analysis. Our code and demo page can be found at https://github.com/keshavbhandari/yinyang. 1 Introduction When developing a short musical motif into a complete monophonic piece, composers can repeat the previous context, vary it slightly, or create something novel departing from the original motif. While composing, it is important to consider local coherence by ensuring that the next notes logically follow the previous ones, in addition to incorporating motivic variations throughout, for an overarching structure. Additionally, one may aim to produce new sections to establish global form. This is common practice in western European musical composition where people employ techniques like fragmentation, sequencing, retrograde, inversion, etc. to develop the musical idea [ 21 ] . However, blindly applying these rules without considering the preceding context could be musically nonsensical. Conversely, straying too far from the original motif can lead to a lack of cohesion and unity. Finally, in some genres, repeating themes too much can be perceived as monotonous by listeners, making it undesirable [ 4 ] . Computationally replicating the compositional process of developing a motif into a structured piece has been researched for decades, but remains an open challenge today [ 3 , 13 ] . Unlike storytelling with clear plot structures, the “language” of musical structure is more abstract and heavily relies on repeating ideas, varying them slightly, and expanding on them. This makes explicitly modelling and representing musical structure challenging. Some studies consider the musical structure to be implicitly learnt during the training process. Approaches such as those in [ 11 , 22 , 34 , 2 ] include various enhancements to the model architecture. Data-based enhancements, where additional structural data annotations are provided to improve the model’s explicit generalizability of structure, are also explored. For example, the Lookback RNN [ 26 ] encodes repeating patterns from 1 or 2 bars ago in the input to allow the RNN model to better identify repeating events in the generated melody. Similarly, the R-Transformer [ 10 ] is trained on a dataset of 5 types of motif repetitions with a multiple loss objective. On the other hand, representation-based enhancements build on the idea that compact sequences would allow the model to see the complete musical work during training, enabling it to learn longer structural dependencies over the entire piece’s duration [ 9 , 12 ] . Increasing the model’s context window to fit much larger sequences is yet another way to expose the complete piece to the model [ 33 , 6 ] . Although these models show improvement over the baselines, they lack flexibility and control as their structure is automatically learnt. Another class of models draws inspiration from the hierarchical nature of musical elements such as notes, chords, bars, phrases, and sections [ 8 , 28 , 36 , 35 ] . These models use this inherent hierarchy by employing specialized networks operating at various scales to generate music. However, the window sizes across the hierarchical tiers of the neural network are fixed during training and may not match the varying hierarchy in music structure across pieces and genres. Pre-training methods have also been explored recently with models such as MelodyGLM [ 32 ] , which introduces a multi-task pre-training framework tailored for melodies and a fine-tuning application for melody continuation and infilling tasks. Similarly, MuPT [ 19 ] explores the application of large language models to the pre-training of melodies encoded with a novel multi-track ABC notation. Recently, MelodyT5 [ 30 ] uses the T5 transformer architecture [ 20 ] for pre-training with multiple objectives on a large and diverse corpus of symbolic music data. The model was evaluated on various tasks including melody generation. More recently, a new research area within deep learning called “sub-task decomposition” has emerged [ 3 ] . This approach breaks down the music generation process into smaller, more manageable steps (typically two). The first step aims to learn structural information from the training data, which is then used to guide the generation in the second step. Several studies have utilized this framework to generate melodies [ 25 , 29 , 37 , 16 , 31 ] . Among them, MeloForm [ 16 ] offers user controllability but is dependent on harmony as it is as an integral part of the neural model’s refinement and is therefore not suitable for monophonic melody datasets like the folk songs used in the work presented here. While our framework consists of three distinct models – a phrase generator, a phrase refiner, and a phrase selector, our approach differs slightly from existing sub-task decomposition works. Rather than having one stage aimed at learning structure and another at generating melodies conditioned on it, our three models are trained independently but used together during generation. Our motivation stems from the observation that transformers can generate locally coherent melodies over short time periods. However, in their pursuit of local coherence, they can sometimes make overly safe choices due to the exposure bias of the student-teacher training setup [ 1 ] , leading to an unstable structure over durations longer than 2-3 minutes. Once stuck in a repetitive loop or after deviating too much from the motif, we have found that they lack the ability to correct themselves for generation of the entire piece. This issue with vanilla autoregressive transformer models may be more pronounced when they are trained on short melodies, such as those found in folk songs, but are required to generate longer sequences during inference, where they tend to struggle with maintaining structural coherence and consistent motivic development. This limitation suggests a need for approaches that allow music composers or arrangers to extend a motif into well-structured, longer sequences, an essential aspect of musical development that can often exceed the scope of a model’s training data. We propose Yin-Yang, a transformer-based framework for developing motifs with long-term structure and controllability. Our framework employs a phrase corruption and refinement training strategy using melodic and rhythmic transformations inspired by