Fine-Tuning MIDI-to-Audio Alignment using a Neural Network on Piano Roll and CQT Representations
Sebastian Murgul, Moritz Reiser, Michael Heizmann, Christoph Seibert · 2025
Read Fine-Tuning MIDI-to-Audio Alignment using a Neural Network on Piano Roll and CQT Representations by Sebastian Murgul, Moritz Reiser, Michael Heizmann, Christoph Seibert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22237.txt.
Introduction
Sebastian Murgul and co-authors present a 2025 computing paper on lining up piano recordings with their MIDI files. A MIDI file holds the notes of a performance in rough time order, and aligning it to audio is a standard step in music data work. Human timing quirks make the match loose at the start.
Their network takes an unaligned piano roll and a spectrogram of the audio as inputs. It then outputs the piano roll shifted into line with the sound. Training data was built by bending real MIDI files to mimic common human timing errors. The model beat the standard warping method by up to 20 percent in accuracy across time windows. Joining the two methods gave a further boost in robustness.
Catalog record
| CREATOR | Sebastian Murgul, Moritz Reiser, Michael Heizmann, Christoph Seibert |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-22237
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fine-Tuning MIDI-to-Audio Alignment using a Neural Network on Piano Roll and CQT Representations Sebastian Murgul, Moritz Reiser, Michael Heizmann, Christoph Seibert Abstract In this paper, we present a neural network approach for synchronizing audio recordings of human piano performances with their corresponding loosely aligned MIDI files. The task is addressed using a Convolutional Recurrent Neural Network (CRNN) architecture, which effectively captures spectral and temporal features by processing an unaligned piano roll and a spectrogram as inputs to estimate the aligned piano roll. To train the network, we create a dataset of piano pieces with augmented MIDI files that simulate common human timing errors. The proposed model achieves up to 20% higher alignment accuracy than the industry-standard Dynamic Time Warping (DTW) method across various tolerance windows. Furthermore, integrating DTW with the CRNN yields additional improvements, offering enhanced robustness and consistency. These findings demonstrate the potential of neural networks in advancing state-of-the-art MIDI-to-audio alignment. Keywords Music Information Retrieval \cdot MIDI-to-Audio Alignment \cdot Neural Network 1 Introduction In the last decade, substantial progress has been made in the field of Music Information Retrieval (MIR) using deep learning techniques. A key downstream task is Automatic Music Transcription (AMT) [ 1 ] , which seeks to convert audio recordings of music into a symbolic music representation like MIDI. Since many of the approaches rely on frame-wise annotations and annotating real world recordings is quite time-consuming, the use of synthesized MIDI data has become popular. While this offers a viable path to obtaining large training datasets, incorporating real-world audio could lead to further improvements. Using automatic MIDI-to-audio alignment can accelerate the annotation process and make real-world audio suitable for training. Dynamic Time Warping (DTW) [ 2 , 3 ] is one of the most established algorithmic methods and is considered the industry standard. DTW has been proposed by Dannenberg and Hu in 2003 and uses dynamic programming to optimize a monotonic alignment path by minimizing the cost between the aligned feature vectors. These feature vectors consist of chroma features derived from the spectrogram of the audio and the piano roll of the MIDI. The original method has been improved by several extensions [ 4 , 5 ] , and it has proven to be extremely efficient for aligning or matching recordings of music to the corresponding MIDI transcriptions. In 2009, Ewert et al. introduced the DLNCO features which improve the temporal accuracy without sacrificing the robustness [ 4 ] . The computational complexity of DTW is quadratic in the input length which becomes a major issue when dealing with longer audio recordings. Therefore, Prätzlich et al. proposed a Memory restricted Multiscale DTW (MrMsDTW) [ 5 ] . Besides the DTW method, Gingras et al. proposed a three-step matcher algorithm based on pitch clustering in 2011 [ 6 ] . In 2014, Chen et al. introduced two more algorithmic approaches [ 7 ] . The first one converts the note sequences into strings and performs a string matching. The second one uses the divide and conquer principle to split the input note sequences into smaller units that are synchronized recursively. Besides the algorithmic approaches, neural networks have been used to improve the feature extraction and the DTW cost calculation. Kwon et al. use an RNN-based music transcription algorithm in order to extract chroma features from an audio’s spectrogram which can be used in a DTW alignment with chroma features extracted from MIDI [ 8 ] . Wang et al. use a triplet pair architecture to learn features from low-level representation using a hinge loss function [ 9 ] . Agrawal et al. propose a Siamese network architecture which performs a feature extraction on the audio spectrograms and then estimates a distance between the two audio inputs [ 10 ] . A preprocessing pipeline incorporating DTW and a pre-trained transcription model is used in [ 11 ] and [ 12 ] to align unaligned real-world transcriptions for fine-tuning. In terms of AMT dataset creation, DTW is used for fine alignment of the MAESTRO dataset [ 13 ] in order to account for any jitter in the audio or MIDI recording. In Pop2Piano, Choi et al. use DTW to synchronize piano arrangements to the corresponding original music video [ 14 ] . One major downside of using these techniques for dataset alignment is that they only consider one complete timeframe at once and cannot perform a frequency band specific alignment. Therefore, it can occur that for example the lower notes of a chord are aligned perfectly, but the higher notes are annotated too early. In this work, a deep learning approach for fine-tuning loosely aligned MIDI annotations, which could be derived from DTW, is introduced. The model is based on a CRNN architecture and is trained to estimate an aligned piano roll given an unaligned piano roll and the performance audio. For training, a suited methodology for creating synthetic unaligned training examples from the well known piano transcription dataset MAPS [ 15 ] is presented. Finally, the synchronization results are evaluated with real world examples as a proof of concept. 2 Model Description The two critical components for our approach are the feature extraction of the two cross-modal inputs and the temporal modelling in order to determine the relation between the unaligned and aligned notes. 2.1 Preprocessing The MIDI files are converted to piano roll representations with pitches ranging from 21 to 108 and a temporal resolution of 100 frames per second. The velocity of the notes within the MIDI files is ignored in the piano roll creation, hence resulting in a binary image. In order to retain the individual note onsets of directly successive notes, small gaps with a width of one frame ( 10\text{\,}\mathrm{ms} ) are inserted before an onset if needed. The input audio signal is resampled to 16\text{\,}\mathrm{kHz} . The CQT is calculated with a hop size of 160 using the librosa library [ 16 ] . To match the input piano roll, a total number of 88 frequency bins with 12 bins per octave is used, starting at a minimum frequency of 27.5\text{\,}\mathrm{Hz} . To highlight the contours in the CQT, the magnitude values are employed with a logarithmic scale. Since the audio and the unaligned MIDI file can be of different durations, the CQT and the piano rolls are zero padded to a common temporal length. 2.2 Network Architecture Piano Roll (N\times 88) Conv2D (16,3\times 3) Conv2D (16,3\times 3) Conv2D (32,3\times 3) Dense (256) CQT (N\times 88) Conv2D (16,3\times 3) Conv2D (16,3\times 3) Conv2D (32,3\times 3) Dense (256) BiLSTM (512) Dense (88) Result (N\times 88) Figure 1: Architecture of the proposed CRNN model for MIDI-to-audio alignment. Two parallel convolutional branches extract spectral and temporal features from the unaligned piano roll and CQT input. The features are concatenated and passed through a BiLSTM layer and dense output layer to estimate the aligned piano roll as a binary activation map over time and pitch. The model architecture is visualized in Figure 1 . It is organized in three blocks: two identical convolutional blocks for the feature extraction in the piano roll and the CQT images and one recurrent block that compares the extracted features in order to estimate an output piano roll. The architecture of