Exploring Classical Piano Performance Generation with Expressive Music Variational AutoEncoder
Jing Luo, Xinyu Yang, Jie Wei · 2025
Read Exploring Classical Piano Performance Generation with Expressive Music Variational AutoEncoder by Jing Luo, Xinyu Yang, Jie Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01582.txt.
Introduction
Jing Luo, Xinyu Yang, and Jie Wei generated expressive classical piano performances from nothing in a 2025 paper. A performance is two arts at once: the composer's notes and the pianist's timing and touch. Their system splits the job the same way, starting from a new token representation that packs metrical structure and expressive nuance into compound words a model can learn.
Two branches train together: a quantized autoencoder branch invents the score-like content, playing composer, while a plain autoencoder branch adds expressive detail, playing pianist. A multiscale encoder reads beat-level context and a transformer decoder emits the compound tokens efficiently. Judges and metrics rated its performances above leading rivals in musical quality, per the authors.
Catalog record
| CREATOR | Jing Luo, Xinyu Yang, Jie Wei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01582
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Classical Piano Performance Generation with Expressive Music Variational AutoEncoder Jing Luo, Xinyu Yang, Jie Wei Abstract The creativity of classical music arises not only from composers who craft the musical sheets but also from performers who interpret the static notations with expressive nuances. This paper addresses the challenge of generating classical piano performances from scratch, aiming to emulate the dual roles of composer and pianist in the creative process. We introduce the Expressive Compound Word (ECP) representation, which effectively captures both the metrical structure and expressive nuances of classical performances. Building on this, we propose the Expressive Music Variational AutoEncoder (XMVAE), a model featuring two branches: a Vector Quantized Variational AutoEncoder (VQ-VAE) branch that generates score-related content, representing the Composer , and a vanilla VAE branch that produces expressive details, fulfilling the role of Pianist . These branches are jointly trained with similar Seq2Seq architectures, leveraging a multiscale encoder to capture beat-level contextual information and an orthogonal Transformer decoder for efficient compound tokens decoding. Both objective and subjective evaluations demonstrate that XMVAE generates classical performances with superior musical quality compared to state-of-the-art models. Furthermore, pretraining the Composer branch on extra musical score datasets contribute to a significant performance gain. I Introduction In many musical genres, particularly Western classical traditions, performance is an indispensable element that transforms composers’ abstract compositional ideas into engaging listening experiences through artistic decisions of performers [ 1 ] . Recent advances in AI-Generated Content (AIGC) have sparked growing interest in automatically generating such expressive performances [ 2 ] . This emerging field sits at the intersection of technological innovation and artistic creation, enabling novel applications ranging from virtual performers to enhanced creative tools for musicians. Within piano performance generation research, two primary tasks have emerged: performance rendering [ 3 , 4 ] and performance generation from scratch [ 5 , 6 ] . The former refers to reproducing expressive parameters (e.g., tempo, timing, dynamics, articulation, and pedaling) for given musical scores, essentially simulating the role of pianists. The latter, more challenging task requires joint prediction of both score elements (e.g. notes, rhythm) and their expressive parameters, which constitutes the primary focus of this study. In this paper, we specifically address the two following critical challenges: music representation for expressive details of classical music and effective modeling frameworks. Firstly, conventional approaches to classical piano music generation predominantly employ MIDI-like representations [ 5 ] , based on the MIDI message protocol with musical token encoding via millisecond intervals. Although these representations faithfully represent most piano music, they omit metrical structures (e.g., beats) [ 7 ] , potentially leading to rhythmically inconsistent outputs. Subsequent studies have proposed alternative representations such as REMI [ 7 ] , Compound Word (CP) [ 8 ] and PerTok [ 9 ] . These methods are primarily adopted for music genres with stable tempos (e.g., popular music), but becomes insufficient for representing classical music performances where nuanced timing variations frequently occur even within individual beats. Secondly, in physical music creation, there is a common division between the roles of composers and performers, particularly in classical music traditions. Composers define the abstract musical ideas in the score, specifying what to play, while performers interpret the score, determining how to play it. Most prior research does not distinguish the roles between composer and performer [ 10 , 6 , 11 ] . In most cases, these methods generate the notes and their expressive parameters with same network architecture. These approaches place a heavy burden on the generative model, often resulting in suboptimal generation performance. To address the two challenges, we first introduce the Expressive Compound Word (ECP) representation, a novel method inspired by the Compound Word (CP) representation [ 8 ] . ECP faithfully encodes an expressive music into a sequence of compound tokens. At each time step, a single token comprises two distinct sets of sub-tokens: four score-related sub-tokens that define the notes and metrical structure, and four performance-related sub-tokens that specify detailed expressive parameters of each note. Then, we propose the Expressive Music Variational AutoEncoder (XMVAE), a novel Seq2Seq based framework designed to generate classical piano performances from scratch. XMVAE employs a two-branch Transformer-based architecture that metaphorically divides the task between a Composer and a Pianist . The Composer branch, a VQ-VAE, operates at the note level to generate the detailed scores and outline the metrical structure. In contrast, the Pianist branch, a vanilla VAE, functions at a higher, whole-song level. It interprets the generated score by learning a global representation of performance style. These two branches are trained jointly, ensuring seamless integration between composition and the generation of expressive parameters. Specifically, we incorporate a multiscale encoder in the encoder blocks of both branches, enabling additional context learning at the beat level. Furthermore, we design orthogonal Transformer decoders for both branches to efficiently decode compound tokens across both the temporal axis and the sub-token axis. We conducted both objective and subjective evaluations on classical piano performance datasets to validate the effectiveness of our proposed methods. Both evaluations consistently demonstrate that our approach outperforms the existing benchmark models across most metrics. Furthermore, through ablation studies, we observed that pretraining the Composer branch of XMVAE on an extra musical score dataset significantly enhances overall performance. II Related Work Prior studies on piano performance generation from scratch typically rely on symbolic format like MIDI rather than audio, as MIDI offers an editable and detailed set of performance parameters that can produce highly realistic musical results. The music representations in the most early studies follow the MIDI standard [ 5 , 10 , 6 ] , where temporal tokens are quantized into millisecond bins, leading to the exclusion of metrical structure. To address this, later studies proposed alternative music representations, such as REMI [ 7 ] and CP [ 8 ] . REMI and CP integrate explicit metrical grids to solve metrical structural modeling and enhance expressive dimensions through performance tokens including Velocity and Tempo. However, the Tempo tokens in both REMI and CP operate at the beat level rather than the note level, leading to their limited applications to capture the nuanced expressiveness of classical music, where timing often vary at the note level. The recently proposed PerTok [ 9 ] addresses performance nuances by introducing microshift tokens, which encode MIDI tick offsets from quantized note positions. However, as a REMI-based representation, PerTok still produces longer token sequences compared to CP-like formats. More critically, its design is focused on fixed-tempo modern genres [ 9 ] , which limits its scalability to classical music that often includes frequent tempo fluctuations and is performed in a rubato style. As a result, the quantized note positions in PerTok often misalign with note onsets defined in classical scores, risking potential distortion of the metrical structure. Performance RNN [ 5 ] and Music Transformer [ 10 ] are two of the most influential early studies on performance generation from scratch. Both studies leveraged vanilla sequential models (RNN or Transformer) as backbone architecture. Transformer-GAN [ 6 ] further enhanced the quality of generated samples by incorporating Generative Adversarial Networks (GANs). PerfVAE [ 11 ] explored the disentanglement of musical content and performance style in classical piano performances using a Variational Autoencoder (VAE)