Pairing Real-Time Piano Transcription with Symbol-level Tracking for Precise and Robust Score Following
Silvan . Peter, Patricia Hu (胡紫漪), Gerhard Widmer · 2025
Read Pairing Real-Time Piano Transcription with Symbol-level Tracking for Precise and Robust Score Following by Silvan . Peter, Patricia Hu (胡紫漪), Gerhard Widmer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05078.txt.
Introduction
Silvan Peter, Patricia Hu, and Gerhard Widmer present in 2025 a music tracking system that follows a live piano performance in a score by working in symbols, not just sound. Older systems warp the audio against an audio form of the score and had hit a plateau. Their system first transcribes the live audio into notes in real time, then tracks those notes against the score with a new symbol level tracker.
Compared with a like audio only system, the mixed approach gave smaller tracking errors and succeeded more often. The authors argue that turning sound into symbols early, even with imperfect transcription, makes the tracking task cleaner. The method may help tools for practice, teaching, and live accompaniment.
Catalog record
| CREATOR | Silvan . Peter, Patricia Hu (胡紫漪), Gerhard Widmer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-05078
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pairing Real-Time Piano Transcription with Symbol-level Tracking for Precise and Robust Score Following Silvan . Peter, Patricia Hu (胡紫漪), Gerhard Widmer Abstract Real-time music tracking systems follow a musical performance and at any time report the current position in a corresponding score. Most existing methods approach this problem exclusively in the audio domain, typically using online time warping (OLTW) techniques on incoming audio and an audio representation of the score. Audio OLTW techniques have seen incremental improvements both in features and model heuristics which reached a performance plateau in the past ten years. We argue that converting and representing the performance in the symbolic domain – thereby transforming music tracking into a symbolic task – can be a more effective approach, even when the domain transformation is imperfect. Our music tracking system combines two real-time components: one handling audio-to-note transcription and the other a novel symbol-level tracker between transcribed input and score. We compare the performance of this mixed audio-symbolic approach with its equivalent audio-only counterpart, and demonstrate that our method outperforms the latter in terms of both precision, i.e., absolute tracking error, and robustness, i.e., tracking success. \CJKencfamily UTF8mc 1 Introduction The task of a real-time music tracking system (also known as a score follower) is to listen to a musical performance and recognize automatically, at any time, the current position in the corresponding musical score. Typically, the system possesses a representation of the score in advance, and maps an incoming real-time audio stream onto this representation to decode the current score position. A system that achieves this task accurately and robustly promises to be useful in a wide range of applications, e.g., performance visualizations, live music subtitling, automatic score displays, or live synchronization between a musician and the score following system. In a real-time music tracker, the representation of the two time series to be aligned – score and live performance – plays a crucial role. Generally, there are two principal approaches: representing both time series in the audio domain and then aligning the two audio streams, or processing them in the symbolic domain, where music is represented as note events or MIDI data. The first approach is more versatile as both audio reference tracks (e.g., previous recordings of a piece) as well as audio inputs (e.g., live recordings of an acoustic instrument) are more readily available than their symbolic representations such as MusicXML scores or MIDI performances. Most methods to date have consequently followed the first approach, treating music tracking as an audio-to-audio alignment problem by synthesizing a MIDI version of the score into an audio file. Harmonic (energy distribution-based) or onset (energy dynamics-based) features are then used to align performance and score using online variants of the online time warping (OLTW) algorithm [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] . Apart from these OLTW-based methods, especially earlier systems have relied on Hidden Markov Models (HMMs) and other Bayesian networks for alignment [ 12 , 13 ] . The second approach, symbol-level tracking, is applied for high-precision scenarios such as automatic accompaniment [ 14 ] . While this domain has seen less research, symbol-level trackers tend to be more precise if successful, though they are more susceptible to errors from performance inconsistencies and reference mismatches [ 14 , 15 ] In this paper, we develop a novel symbol-level tracker based on online time warping (OLTW) that incorporates path-wise tempo estimation. We evaluate this symbol-level tracker in a score following scenario where the real-time input consists of an audio recording and the score is provided in a symbolically encoded format. Specifically, we demonstrate that transforming the incoming performance stream from the audio into the symbolic domain, and subsequently aligning it with its corresponding symbolic score, in real time, can surpass the performance of existing audio-based music tracking systems, even when the audio-to-symbolic conversion is imperfect. We compare our method to its online audio-only counterpart and analyze how various tuning parameters, from feature selection to alignment algorithms, influence the performance. Our experimental results highlight the potential of approaching real-time music tracking as a symbol-level task, supported by both improved, reliable symbol-level tracking and recent advances in real-time audio-to-note transcription. 2 Related Work Most existing systems have treated real-time music tracking as an online audio alignment problem, and developed their approaches from offline dynamic time warping (DTW). In (offline) DTW, the two sequences to be aligned are completely known beforehand, and the goal is to find an optimal alignment between the two based on a local distance measure and an alignment cost matrix. Each entry in the matrix represents the cost of the optimal alignment up to that point, and the optimal warping path is found by recursively tracing back through the matrix using dynamic programming [ 16 ] . In the standard version of online time warping (OLTW) [ 1 ] , only the score is known completely beforehand while the performance is streamed in real time. The alignment is now computed incrementally by expanding the window into the (row and/or column) direction with minimal normalized cost. This online variant achieves linear time and space complexity, as it updates only a fixed number of cells per step rather than the entire matrix. Variants of OLTW include enhancements such as the ‘backward-forward strategy’ [ 3 ] that revisits past decisions to improve the precision of the current score position estimate, the use of tempo models [ 5 ] to adjust the score representation based on the tempo of the live performance, as well as the use of multiple reference performances [ 6 , 14 ] . Most audio-based music tracking systems rely on carefully crafted audio features. Widely used features include chroma-based harmonic features [ 3 , 5 , 17 , 18 ] , onset-based features [ 19 ] , or a combination of both, which balance chroma robustness with temporal accuracy [ 2 ] . More recently, the focus has shifted toward learned features, leveraging deep neural networks [ 20 , 21 , 22 , 23 , 9 , 7 , 11 ] . Compared to audio-based music tracking, symbolic music tracking remains relatively under-explored [ 15 , 14 ] . In offline audio-to-score alignment, there is prior work that follows a similar approach to ours, that is, leveraging automatic music transcription (AMT) to predict note presence and onset timing, which are then used as note features for subsequent MIDI-based offline score alignment [ 24 , 25 ] . Moreover, some studies have explored a middle ground between audio and symbol-level alignment by aligning raw transcription features [ 26 ] . Recent advancements in AMT have been largely driven by the availability of large-scale, well-aligned audio-MIDI datasets and increased availability of computational resources [ 27 ] , enabling deep neural networks to achieve state-of-the-art performance [ 28 , 29 ] . However, as these models have grown in complexity and size, they have become increasingly resource-intensive, limiting their applicability in real-time scenarios. Most existing systems are therefore designed for offline use, with only a limited number addressing the task in real-time contexts [ 30 , 31 , 32 ] . Online methods for