ImprovNet - Generating Controllable Musical Improvisations with Iterative Corruption Refinement Thanks: This work was supported by UKRI and EPSRC (grant EP/S022694/1) and by SUTD’s Kickstart Initiative (grant SKI 2021_04_06). The training carried out in this research utilized AI Industrial Convergence Cluster supported by the Ministry of Science and ICT of Korea, and Gwangju Metropolitan City. Keshav Bhandari 1 , Sungkyun Chang 1 , Tongyu Lu 2 , Fareza R. Enus 1 , Louis B. Bradshaw 1 , Dorien Herremans 2 , Simon Colton 1 Abstract Despite deep learning’s remarkable advances in style transfer across various domains, generating controllable performance-level musical style transfer for complete symbolically represented musical works remains a challenging area of research. Much of this is owed to limited datasets, especially for genres such as jazz, and the lack of unified models that can handle multiple music generation tasks. This paper presents ImprovNet, a transformer-based architecture that generates expressive and controllable musical improvisations through a self-supervised corruption-refinement training strategy. The improvisational style transfer is aimed at making meaningful modifications to one or more musical elements - melody, harmony or rhythm of the original composition with respect to the target genre. ImprovNet unifies multiple capabilities within a single model: it can perform cross-genre and intra-genre improvisations, harmonize melodies with genre-specific styles, and execute short prompt continuation and infilling tasks. The model’s iterative generation framework allows users to control the degree of style transfer and structural similarity to the original composition. Objective and subjective evaluations demonstrate ImprovNet’s effectiveness in generating musically coherent improvisations while maintaining structural relationships with the original pieces. The model outperforms Anticipatory Music Transformer in short continuation and infilling tasks and successfully achieves recognizable genre conversion, with 79% of participants correctly identifying jazz-style improvisations of classical pieces. Our code and demo page can be found at https://github.com/keshavbhandari/improvnet. I Introduction Music style transfer has recently emerged as a fascinating area of generative AI research, offering new possibilities for creating personalized compositions. Altering the melodic, rhythmic, and harmonic aspects of compositions can enable the crafting of music that feels both familiar and creative, tailored to reflect individual preferences or to evoke specific artist and genre styles. The concept of neural style transfer was pioneered by [ 1 ] , with the goal of blending “content” from one source with the “style” characteristics of another. In this framework, content refers to the underlying structure that remains intact, while style encompasses the features transferred from a different input. Since then, neural style manipulation has sparked widespread interest in various modalities. However, the core idea behind style transfer remains consistent across these applications: preserving the content or structure of one input while adopting the stylistic characteristics of another. In recent years, the concept of “style transfer” has evolved, particularly in the realm of music processing. As seen in [ 2 , 3 , 4 , 5 , 6 ] , the definition now encompasses a wider and more general range of style transformations rather than deriving it from one or a few examples. In this context, the goal is to transform an input to match the stylistic characteristics of large datasets. According to [ 7 ] , this can also be described as style conversion or translation . In our work, we adhere to the style conversion definition from the recent literature. In this paper, we focus on the generation of expressive performance improvisations in the context of symbolically represented classical and jazz solo piano pieces. An improvisation could be viewed as a special case of style conversion where the generated output either adopts a different style or retains the original one through meaningful modifications to the given content. The goal of improvisation is to introduce changes that maintain a recognizable link to the original, preserving its identity while exploring new variations within its stylistic representations. Hence, the musical content is generated with the aim of reflecting the essence of the original, either within the same style or a distinct one, depending on the user. We interpret “style” throughout this paper as genre-specific elements, such as rhythmic, harmonic, and melodic properties that reflect a particular musical genre. On the other hand, “content” encompasses the foundational elements specific to a musical piece, including its core melody, rhythm, chord progressions, and structure. According to this, our goal for cross-genre music improvisation is to adapt the content of a music piece to reflect the stylistic musical attributes representative of the target genre while preserving the core musical ideas of the original. For example, a classical to jazz cross-genre improvisation may preserve motivic fragments, tempo, and global form of the original piece while introducing elements such as dissonance, chromatic scales, irregular or syncopated rhythms, jazz harmonies, and melodies. Similarly, our goal for intra-genre improvisation (classical-to-classical or jazz-to-jazz improvisation) is to adhere to the genre of the original piece while modifying its content, generating a new variation that still references the essence of the initial composition. Our study addresses a gap in symbolic music research in generating controllable expressive performance style transfer for complete musical works [ 8 ] . This gap arises from challenges such as limited symbolic data, especially in jazz [ 9 ] , the scarcity of expressive polyphonic solo piano jazz datasets, and the lack of well-annotated datasets [ 10 ] . These issues hinder the training of generative models capable of capturing stylistic variations and performance dynamics. Furthermore, ambiguities in the definition of target genres [ 11 ] reduce user control over transformations, leading to low-fidelity outputs. Finally, no single symbolic music model integrates tasks like variation creation, harmonization, infilling, and continuation within a unified framework, an essential step to advance style transfer and enabling personalized music generation. To address these challenges, we propose ImprovNet, a transformer-based architecture [ 12 ] that is pre-trained and fine-tuned with a self-supervised corruption refinement infilling strategy . ImprovNet is designed to generate expressive and controllable improvisations for complete musical pieces. The improvisational style comes from modifications to one or more musical elements - melody, harmony, or rhythm of the original composition to emulate the spontaneity and creativity of human improvisation. The extent to which the stylistic elements get modified dictates how recognizable the original content is. In our approach, we allow the user control over the transformation functions and the degree of transformation applied to the musical elements, letting them choose whether the generation remains highly recognizable or takes on a more creative reinterpretation. Due to the nature of the corruption refinement infilling task, as we explain below in the methods section, ImprovNet is also capable of harmonizing a melodic line, performing short prompt continuation and short infilling tasks in addition to generating style-aware improvisations. II Related Work Music style transformations come in various forms, depending on how style is defined, how music is represented, and what the model is conditioned on. In this paper, we focus on expressive and controllable improvisational style in the symbolic domain, achieved through modifications to the rhythm, harmony, and melody of a solo piano performance conditioned on the target genre. Previous studies have focused on these tasks separately. For example, [ 13 ] addresses style transfer for long pop piano pieces, allowing users to customize musical attributes such as rhythmic intensity and polyphony at the bar level. Models such as [ 14 , 15 ] harmonize a melody in the style of Bach chorales. Similarly, [ 16 ] proposes FolkDuet to fuse Chinese melodies with musical counterpoint in real time. [ 17 ] introduces FaderNets for generating musical variations by adjusting low-level attributes using latent regularization and feature disentanglement while enabling high-level feature control (e.g., arousal) through a VAE framework. [ 18 , 19 ] focus on arranging accompaniments for solo piano melodies, offering control over aspects such as voice density. Studies such as [ 20 , 21 , 22 , 23 , 24 , 7 ] extend the controllable accompaniment arrangement task to multi-track generation. However, all of these approaches use highly quantized piano encodings, limiting the model’s ability to produce expressive renditions. An expressive performance rendition in symbolic music should be capable of human-like characteristics in timing, dynamics, and articulations, which go beyond the rigid quantization of notes. [ 25 ] was among the first works to generate expressive performance renditions by modelling a stream of midi note events. Recently, generating expressive performance renditions from symbolic score has been the focus of several works such as [ 26 , 27 , 28 , 29 , 30 , 31 , 32 ] . Among these, [ 31 ] is capable of generating controllable and expressive polyphonic piano variations, but is limited to short fragments of music. Several works have explored genre style conversion, which is more closely related to this paper. [ 2 ] explores unsupervised monophonic melody style conversion using a statistical framework that integrates a music language model with an edit model, discovering musical styles through unsupervised grammar induction. [ 33 ] pioneered the use of GANs for genre conversion in symbolic polyphonic piano pieces by adapting the CycleGAN [ 34 ] framework with genre classifier discriminators. [ 35 ] enhanced this approach by incorporating a beta-VAE [ 36 ] for better disentanglement of style and content. Recently, [ 37 ] and [ 38 ] have advanced multi-instrument genre conversion through architectural and loss function-related modifications to the CycleGAN and VAE frameworks. However, these GAN-based approaches face training instability, potential mode collapse, and challenging parameter tuning. Their piano-roll representations also limit expressive performance capabilities and lack user-controlled style conversion intensity, both key aspects of our work. Recent works on infilling and prompt continuation tasks, including [ 24 , 20 , 39 , 40 , 41 ] , also relate to our model’s capabilities. III Methods III-A Tokenization Fig. 1 : A 10 second piano roll depicting the Aria tokenizer that uses absolute onsets relative to the start of the segment. We adapt a publicly available tokenizer called Aria 1 1 1 https://github.com/EleutherAI/aria-utils to encode expressive piano performances for ImprovNet. Although the Aria tokenizer can encode all MIDI instrument categories, we restrict it to solo piano and additionally change the order of tokens to assist with the harmonization task for which the generated logits are constrained, as shown later in section III-F4 . As shown in Fig. 1 , the Aria tokenizer uses a chunked absolute onset encoding. In this encoding, the pitch and velocity of the MIDI are merged into a single token, while the onset and duration remain as separate tokens. Most metadata, such as bars, key, and time signatures, is excluded. However, essential musical elements such as the usage of the sustain pedal are directly incorporated into the onset and duration tokens. Quantization is minimal: onset and duration values are rounded to the nearest 10 milliseconds, and velocity values are quantized to increments of 15 MIDI units (ranging from 0 to 120). The use of minimally quantized absolute onsets allows the music to be encoded in a manner that closely preserves the characteristics of a human-like performance rendition. One disadvantage of using absolute onsets is that the vocabulary size scales with the length of the piece. This is problematic as transformer models struggle with arithmetic [ 42 ] . To address this issue and avoid sparse tokens during training, the Aria tokenizer chunks the sequence into 5,000 millisecond segments, separating each chunk with a special token \langle T\rangle . The \langle T\rangle token resets the absolute onsets to start from 0, similar to the function of the ‘bar’ token in the REMI [ 43 ] encoding, thus controlling the vocabulary size. Fig. 2 : A) A random segment from S is selected with context segments around it. B) This segment is corrupted with a corruption function. C) ImprovNet refines it back. Fig. 3 : A single pass iterates over all the segments in S . Segments marked with * are preserved. The other segments are corrupted with the corruption function and refined back. III-B ImprovNet Overview ImprovNet relies on a corruption-refinement training strategy similar to [ 44 ] , to generate music in an improvisational style of the original composition. The entire piece is segmented into 5-second segments as seen in Fig. 1 . During training (Fig. 2 ), a random segment is selected with context segments on its left and right. This segment is corrupted using one of the corruption functions described in Section III-D . The corrupted segment, combined with the original context, is fed into a transformer encoder along with conditional tokens for the genre and corruption type. The transformer decoder learns to reconstruct the original segment. During generation (Fig. 3 ), 5-second segments are processed iteratively. A chosen segment is corrupted and refined based on a probability similar to that of training. For cross-genre improvisation, refinement conditions on the target genre token. This process iterates from beginning to end several times, gradually shaping the piece into the target genre. III-C Formulation of Refinement Framework Let S denote all 5-second segments separated by the \langle T\rangle tokens. We can represent the encoding for a complete musical sequence as: S=(s_{1},\dots,s_{N}) (1) During training, we randomly sample a segment s_{i} for i