Velocity Prediction in Automatic Guitar Transcription

Jackson Loth, Xavier Riley, Simon Dixon and Emmanouil Benetos · 2026

Read Velocity Prediction in Automatic Guitar Transcription by Jackson Loth, Xavier Riley, Simon Dixon and Emmanouil Benetos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24912.txt.

Introduction

Jackson Loth, Xavier Riley, Simon Dixon, and Emmanouil Benetos teach a guitar transcription model to predict how hard each note is played. Velocity, the intensity of a note, is rarely predicted. Datasets lack such labels, and the notion lacks a clean definition away from the piano. The team's workaround is synthetic data. Virtual instruments generate guitar audio where every note's strength is known by construction.

A model is pretrained on that synthetic data to learn the skill. Its weights then transfer into a second model trained on real guitar audio. The transferred model keeps the velocity skill while gaining real world accuracy and generalisation. On synthetic tests it beats a baseline lacking the pretrained weights. Note transcription also improves slightly, though the gain is small and not always significant. Overall performance matches the state of the art, with the new skill added on top.

Catalog record

CREATORJackson Loth, Xavier Riley, Simon Dixon and Emmanouil Benetos
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24912

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Velocity Prediction in Automatic Guitar Transcription Thanks: This work is supported by Innovate UK (Project no. 10105279) and the UKRI Centre for Doctoral Training in Artificial Intelligence and Music (Grant no. EP/S022694/1). This research utilised Queen Mary’s Apocrita HPC facility, supported by QMUL Research-IT. http://doi.org/10.5281/zenodo.438045 Jackson Loth, Xavier Riley, Simon Dixon and Emmanouil Benetos Abstract Automatic Music Transcription (AMT) models have achieved a high level of success in polyphonic transcription of various instruments. Velocity, typically a measure of note intensity, is less commonly predicted in these models due to the absence of velocity labels in available datasets and lack of a proper definition for instruments other than piano. We present a methodology and model for velocity prediction in Automatic Guitar Transcription (AGT) which uses virtual instruments to generate synthetic training data with velocity labels. We first pretrain a model on this synthetic data. These weights are then transferred to a different model and trained on real guitar audio, allowing the model to retain the working velocity prediction while also achieving high performance and generalisability from the real training data. The velocity prediction is shown to outperform a baseline model which does not use the pretrained velocity weights, when evaluated on synthetic data. In addition, using the pretrained velocity weights offers a small improvement in note transcription, though the magnitude of this improvement is limited and not always significant depending on the testing data. Overall the model achieves results comparable to the state of the art in guitar transcription, while also successfully predicting velocity. I Introduction Automatic Music Transcription (AMT) involves transcribing musical audio to some symbolic format such as MIDI [ 1 ] . Like many music information retrieval (MIR) tasks, it has found great success in recent years through the use of deep learning. While research has explored a variety of different instruments, a small dedicated subfield has grown around the guitar. Commonly referred to as Automatic Guitar Transcription (AGT), progress in this task has been steadily increasing in recent years. However, focusing on guitar can be challenging due to the relative difficulty of obtaining high-quality data compared to other instruments such as piano. To address this, several datasets of guitar audio have been created and made available by researchers, such as GuitarSet [ 24 ] , IDMT-SMT-GUITAR [ 10 ] , EGDB [ 2 ] , GAPS [ 21 ] , Guitar-TECHS [ 19 ] and GOAT [ 16 ] . These datasets all provide note-level and sometimes string-level annotations with real guitar audio, but are also unfortunately difficult and time-consuming to create. This difficulty can however be offset somewhat through score alignment techniques [ 17 , 22 , 26 ] by automatically aligning scores with audio, allowing annotations to be more easily created without specialised hardware such as hexaphonic pickups. While these works achieve impressive results in pitch transcription, velocity tends to be ignored by AGT papers. Velocity is a somewhat ambiguous and abstract value in the context of guitar, despite being an important component of expressiveness in a musical performance. While it is generally thought of as an indication of dynamic level or loudness, it tends to have a nonlinear relationship with loudness [ 3 ] and is perhaps better thought of as a general measure of “intensity”, related to both loudness and timbre. The values that are controlled by velocity in virtual instruments are left more to the developer of the instrument rather than following any kind of standard. Given this vague definition of velocity, it is unsurprising that, outside of piano transcription where velocity is actually measured with a Disklavier [ 6 , 14 ] , velocity is often ignored in favor of focusing on transcribing each note’s pitch, onset and offset. Datasets such as MAESTRO [ 7 ] , Studio MAESTRO [ 4 ] and Saarland [ 18 ] , which were created using a Disklavier, enable easier training and evaluation of piano velocity transcription. While DSP-based approaches to velocity prediction exist [ 5 ] , most recent papers focus on deep learning-based approaches [ 11 , 12 , 13 , 8 , 9 ] . Given these difficulties with velocity and the absence of any datasets which contain proper velocity annotations for guitar, we instead use the velocities defined by guitar virtual instruments to create a synthetic dataset which we use to train our model. These velocity labels, which broadly map to volume and timbre in the virtual instrument, are used to define velocity in the context of this model. Once we train this initial velocity model, we can integrate the trained velocity prediction into a separate model trained on real-world audio, giving us the velocity prediction along with the accuracy of a model trained on high quality real data. The approach of using synthetic data to train AGT models has been shown to be effective in note onset and offset prediction [ 15 ] , though this approach has not yet been applied to guitar velocity prediction. Prior work has however shown promise in a similar approach for predicting MIDI expression values in wind and string instruments [ 25 ] . The contents of this paper are as follows: (1) an overview of the data collection, training and evaluation methodology; (2) an evaluation of the velocity prediction; and (3) an evaluation of the effects of velocity prediction on the note transcription. To our knowledge this is the first work to extend guitar transcription to also include velocity prediction. Fig. 1: Visualization of the training methodology. II Methodology II-A Data To train the velocity prediction part of our model, we use a synthetic dataset. While publicly available synthetic datasets of guitar audio such as SynthTab [ 27 ] exist, they do not contain suitable velocity annotations. Our data was instead created using the FrançoisLeduc dataset [ 22 ] , a collection of 79 aligned real guitar audio and MIDI annotations spanning four hours. For each note in the MIDI annotations, we calculated a rough estimate of the velocity. The estimated velocities were obtained by first isolating the audio corresponding to the onset and offset of each note. For each note segment, the audio was transformed into a frequency representation and filtered using bandpass filters centered at the note’s fundamental frequency and harmonics. We then computed the total energy of each filtered signal and applied a weighted sum with exponentially decaying weights for each harmonic. This was normalised to a range of 0 to 127 to convert to velocity values. This estimated velocity information was applied to the corresponding MIDI annotations and rendered using five different timbres from three guitar virtual instruments 1 1 1 Native Instruments Session Guitarist: Picked Nylon ; Native Instruments Session Guitarist: Picked Acoustic ; Straight Ahead Samples JB-145: Archtop Guitar , creating roughly 20 hours of synthetic training data with velocity. Through this process, we were able to obtain synthetic guitar audio in which the performances correspond exactly to the velocity values in the performance MIDI. The velocity curve used by these virtual instruments (the level/timbre/intensity which corresponds to each velocity value) defines the “true” velocity curve which our models learn to predict. In addition, we use a variety of real audio datasets in training and evaluation. The

Read the complete text