Aligned Music Notation and Lyrics Transcription Journal: Pattern Recognition

Eliseo Fuentes-Martínez, Antonio Ríos-Vila, Juan C. Martinez-Sevilla, David Rizo, Jorge Calvo-Zaragoza · 2024

Read Aligned Music Notation and Lyrics Transcription Journal: Pattern Recognition by Eliseo Fuentes-Martínez, Antonio Ríos-Vila, Juan C. Martinez-Sevilla, David Rizo, Jorge Calvo-Zaragoza (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04217.txt.

Introduction

Eliseo Fuentes-Martínez, Antonio Ríos-Vila, Juan Martinez-Sevilla, David Rizo, and Jorge Calvo-Zaragoza published a paper in 2024 defining a new transcription problem for digitizing sung music. A choir score is not just notes and not just words: each syllable belongs to a specific note, and losing that pairing ruins the edition. Standard music and text recognition tools handle the halves separately.

The authors formalize the joint task and compare strategies, from splitting the problem into pieces to end-to-end systems that read the whole score at once, including language-model approaches. Four datasets of Gregorian chant, real and synthetic, plus new scoring metrics, support the tests. End-to-end methods generally beat the piecemeal ones at keeping notes and lyrics aligned, with language models strongest when training data abounds.

Catalog record

CREATOREliseo Fuentes-Martínez, Antonio Ríos-Vila, Juan C. Martinez-Sevilla, David Rizo, Jorge Calvo-Zaragoza
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04217

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Aligned Music Notation and Lyrics Transcription Journal: Pattern Recognition Eliseo Fuentes-Martínez, Antonio Ríos-Vila, Juan C. Martinez-Sevilla, David Rizo, Jorge Calvo-Zaragoza Abstract The digitization of vocal music scores presents unique challenges that go beyond traditional Optical Music Recognition (OMR) and Optical Character Recognition (OCR), as it necessitates preserving the critical alignment between music notation and lyrics. This alignment is essential for proper interpretation and processing in practical applications. This paper introduces and formalizes, for the first time, the Aligned Music Notation and Lyrics Transcription (AMNLT) challenge, which addresses the complete transcription of vocal scores by jointly considering music symbols, lyrics, and their synchronization. We analyze different approaches to address this challenge, ranging from traditional divide-and-conquer methods that handle music and lyrics separately, to novel end-to-end solutions including direct transcription, unfolding mechanisms, and language modeling. To evaluate these methods, we introduce four datasets of Gregorian chants, comprising both real and synthetic sources, along with custom metrics specifically designed to assess both transcription and alignment accuracy. Our experimental results demonstrate that end-to-end approaches generally outperform heuristic methods in the alignment challenge, with language models showing particular promise in scenarios where sufficient training data is available. This work establishes the first comprehensive framework for AMNLT, providing both theoretical foundations and practical solutions for preserving and digitizing vocal music heritage. 1 Introduction The main obstacle to carrying out digital musicology tasks on a large scale is the transcription of written musical sources into a format that can be further processed by a computer [ 25 ] . This transcription process is costly when done manually, as the complexity of music notation requires the use of specialized and hard-to-manage music score editors, along with expert supervision. The challenge becomes even more discouraging for historical music notation systems, for which suitable tools might not exist. Consequently, automatic transcription systems for music documents are invaluable [ 4 ] . OMR ( OMR ) is a field of computer science dedicated to reading music notation from document images [ 7 ] . It has been an active research area for decades [ 8 ] . Typically, the output of an OMR system is a structured digital format, such as MusicXML or MEI, which encodes the musical content for further processing. Traditionally, OMR systems focused on the detection and recognition of music symbols using heuristic image processing techniques [ 27 ] . However, deep learning brought about a paradigm shift [ 35 , 24 , 41 ] , opening new possibilities to advance the field that were once considered infeasible. One such task is the automatic transcription of vocal music documents. Vocal music refers to compositions where the singing part is central to the piece, whether accompanied by instruments or not. Thus, an OMR system for this type of music must handle not only the transcription of the music notation but also the lyrics that indicate the words to be sung. Both modalities represent complementary aspects of the same musical work: the text specifies “what” to sing, while the music notation specifies “how” to sing it (see Fig. 1 ). Figure 1: Vocal music excerpt. It is important to mention how, in this type of music, there is no one-to-one relationship between musical notes and lyrics; it is rather a many-to-many relationship, thus making the alignment so crucial to know how these scores should be interpreted. While OMR algorithms can independently recognize the music notation [ 5 ] and text recognition algorithms can handle the lyrics [ 21 ] , such approaches fail to address the alignment between notes and lyrics—a critical requirement for meaningful musicological outcomes. This challenge is particularly interesting from a scientific perspective, as there is no existing deep learning framework for it. This paper is the first in the field of OMR to comprehensively address the transcription of vocal scores. We achieve this by formally defining the AMNLT ( AMNLT ) challenge, which emphasizes not only the transcription of music notation and lyrics but also the critical alignment task (see Fig. 2 ). We also analyze existing divide-and-conquer approaches and propose how end-to-end solutions can be adapted to include alignment information for music scores. Additionally, this paper introduces a set of metrics to assess the performance, evaluating both transcription and alignment accuracy. All these aspects are evaluated through experimentation on four AMNLT scenarios, comprising three real datasets and one hybrid dataset. The results demonstrate that (i) divide-and-conquer methods, while precise in transcription, fail to provide complete results for AMNLT, and (ii) the end-to-end approaches outperform traditional methods in both transcription quality and alignment precision, establishing a new baseline for full AMNLT. Figure 2: Graphical representation of the AMNLT framework, depicting its three key components: music notation (turquoise), lyrics (magenta), and their alignment (purple). The remainder of the paper is structured as follows: Section 2 reviews how the OMR literature addresses the transcription of music and lyrics, highlighting overlooked aspects. Section 3 provides a formal definition of the AMNLT challenge. Section 4 discusses the adaptations required for state-of-the-art transcription pipelines to address AMNLT and introduces two holistic methods to address this challenge. In Section 5 , we describe the case study and present the four datasets used in our experiments, while Section 6 explains the metrics developed to evaluate the performance of the models. Section 7 details the implementation of the proposed approaches. The results are discussed in Section 8 , and the paper concludes in Section 9 . 2 Background OMR methods typically begin with layout analysis, where the document is divided into distinct regions of interest such as staves, lyrics, titles, and other elements. This step is essential for isolating the structural components of a music score. Current layout analysis methods for music scores are capable of robust and accurate region extraction, even under challenging conditions [ 40 , 12 ] . Once the regions are extracted, modern systems predominantly adopt end-to-end methods to retrieve the content within each region in a single step [ 11 ] . This formulation offers significant advantages over traditional symbol-based pipelines by learning contextual relationships directly from data. Typically, these methods are based on Convolutional Recurrent Neural Networks (CRNN) combined with CTC ( CTC ) loss [ 10 , 3 , 24 ] , although some authors have recently incorporated the use of Transformers [ 28 ] . This end-to-end approach has achieved notable success across various types of musical sources, including both printed and handwritten documents, and can be considered the state of the art for OMR . For tasks involving vocal music, the transcription challenge extends beyond recognizing music notation to include lyrics transcription, which is traditionally handled independently [ 6 ] . While it is possible to reliably extract music and lyrics information separately, this approach fails to address the critical alignment between the two modalities. As mentioned above, this alignment is essential for interpreting vocal music, as it synchronizes the “what” (lyrics) with the “how” (music notation). This concept of alignment has been widely studied in other domains, such as the automatic transcription of music from audio recordings [ 17 , 32 , 30 ] , yet it remains largely unexplored in the context of document image analysis for vocal music. The literature reveals only

Read the complete text