LEGATO: Large-scale End-to-end Generalizable Approach to Typeset OMR

Guang Yang, Victoria Ebert, Nazif Tamer, Luiza Pozzobon, Noah A. Smith · 2025

Read LEGATO: Large-scale End-to-end Generalizable Approach to Typeset OMR by Guang Yang, Victoria Ebert, Nazif Tamer, Luiza Pozzobon, Noah A. Smith (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19065.txt.

Introduction

Guang Yang and co-authors present Legato, a model that reads full pages of printed sheet music and writes them out in a text notation, in a 2025 paper. Optical music recognition is hard because scores pack many tiny symbols across a page. Legato is the first large pre-trained model for typeset scores and the first to output ABC notation, a compact text format for music.

A vision encoder feeds a decoder trained on more than 214,000 score images. Across several test sets the model beat the prior best by wide margins, cutting standard error scores by about 68 and 48 percent on the most realistic set. The authors credit scale and the text-based output for its reach across varied printed styles.

Catalog record

CREATORGuang Yang, Victoria Ebert, Nazif Tamer, Luiza Pozzobon, Noah A. Smith
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-19065

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LEGATO: Large-scale End-to-end Generalizable Approach to Typeset OMR Guang Yang, Victoria Ebert, Nazif Tamer, Luiza Pozzobon, Noah A. Smith Abstract We propose Legato, a new end-to-end transformer model for optical music recognition (OMR). Legato is the first large-scale pretrained OMR model capable of recognizing full-page or multi-page typeset music scores and the first to generate documents in ABC notation, a concise, human-readable format for symbolic music. Bringing together a pretrained vision encoder with an ABC decoder trained on a dataset of more than 214K images, our model exhibits the strong ability to generalize across various typeset scores. We conduct experiments on a range of datasets and demonstrate that our model achieves state-of-the-art performance. Given the lack of a standardized evaluation for end-to-end OMR, we comprehensively compare our model against the previous state of the art using a diverse set of metrics. 1 Introduction A substantial portion of written music exists only as photocopies of printed sheet music (e.g., IMSLP; Project Petrucci LLC 2025 ). Digitalizing these images into modern, machine-readable symbolic formats would enable data for a wide range of music analysis and synthesis applications. With this goal in mind, we tackle the problem of optical music recognition (OMR) to efficiently convert images of typeset scores to symbols. Recent approaches to this problem have proposed end-to-end OMR systems ( Ríos-Vila et al. 2024 ; Ríos-Vila et al. 2024 ; Ríos-Vila et al. 2023 ; Mayer et al. 2024 ; Calvo-Zaragoza & Rizo 2018b ) , focusing exclusively on formats for piano, monophonic music, or single-system scores. More generalizable solutions require careful consideration of the diversity of inputs (i.e., complex layouts that contain multiple systems, staves and voices on a single page, as well as extensive text annotations such as titles and lyrics) and outputs (each output format—e.g., MusicXML, ABC, **kern—has its own advantages for OMR and current evaluation heavily relies on output format choice). Our main contributions are as follows. • We construct a new multi-page large-scale OMR dataset , PDMX-Synth, rendered from symbolic scores in PDMX ( Long et al. 2025 ; Xu et al. 2024 ) with diverse rendering schemes (§ 3 ). • We introduce Legato, the first end-to-end OMR model built upon a pretrained vision encoder, and capable of recognizing multi-page typeset scores (§ 4 ). • Legato achieves state-of-the-art performance on multiple OMR datasets, including a newly constructed sample from IMSLP ( Project Petrucci LLC 2025 ) , with a large improvement over the previous best model (§ 6 ). We release the code to reproduce our work at https://github.com/guang-yng/legato . 2 Baseline and Design Considerations Figure 1: Model architecture. The input image is first cropped into overlapping segments with an aspect ratio of 1:4 or less, then resized and divided into four patches (§ 4.2.1 ). The image patches are fed into a vision encoder (§ 4.2.2 ; parameters are frozen during training). The resulting latent embeddings serve as cross-attention keys and values in a transformer decoder, which autoregressively generates ABC tokens (§ 4.2.3 ). Special tokens <B> , <I> , and <E> denote <|begin_of_abc|> , <|image|> , and <|end_of_abc|> , respectively. For better visualization, here we use “_” to represent whitespace. 2.1 Multi-System OMR Our starting point is the success of Sheet Music Transformer++ (SMT++; Ríos-Vila et al. 2024 ), which is, to our knowledge, the only model designed to handle multiple systems in a score rather than single-staff or single-system scores ( Mayer et al. 2024 ; Ríos-Vila et al. 2023 ) . SMT++ is an encoder-decoder transformer model trained end-to-end on purely synthetic, full-page piano-form scores. It was trained on FP-GrandStaff (688 pages), which was generated by randomly concatenating single-system piano scores from GrandStaff ( Ríos-Vila et al. 2023 ) . SMT++ is the baseline against which we compare our approach. 2.2 Output Score Representation Many different symbolic score formats have been considered in past work on OMR; this paper considers **kern, ABC, and MusicXML. **kern is a musical representation designed within the Humdrum toolkit ( Huron 1997 ) . **kern is ASCII-based and places pitch and relative duration as the main focal points of the format, with visual information coming in second. **kern explicitly models concurrent notes using spaces for notes in the same staff, and tabs for notes in other staves. It was designed with music researchers in mind, and with some study can be fluently read by humans. **kern is an OMR target in SMT++ ( Ríos-Vila et al. 2024 ) and other systems ( Ríos-Vila et al. 2024 ; Ríos-Vila et al. 2023 ) . The ABC music standard was introduced as an alternative ASCII-based form of musical notation ( Walshaw 2011 ) . Similar to **kern, it can be written and read by any text editor. The main attraction of ABC notation is the simple, concise format—a song that would take thousands of lines in other formats will take only tens in ABC, potentially reducing the computational cost for autoregressive models like our decoder. ABC notation can encode lyrics, title, tempo, decorations, articulations, and even some kinds of typesetting parameters, making it a nearly comprehensive format. The more explicit structure for musical engravings such as barlines and linebreaks, as well as a notation more similar to sheet music than that of **kern, has led to ABC notation as a target for many systems ( Wu et al. 2024 ; Wu & Sun 2023 ; Casini et al. 2024 ) , but to our knowledge, image-to-ABC OMR models have not yet been trained. MusicXML is a tree-based music notation format introduced in 2001 ( Good 2001 ) , widely adopted by commercial software such as MuseScore ( MuseScore Ltd. 2021 ) and Sibelius ( Avid Technology 2025 ) . Its popularity has made it a common target for OMR and music transcription tasks ( Beyer & Dai 2024 ) , sometimes with simplifications ( Mayer et al. 2024 ) . However, compared to ABC and **kern, MusicXML is more verbose, harder to parse, and its hierarchical structure poses challenges for sequence models. Each of these formats, despite their differences, carry much of the same information. They all include some intrinsic method of codifying visual score information such as barlines and phrasing, and each encodes note length as an absolute value, similar to typical human-readable sheet music—all elements that other popular symbolic music formats, such as MIDI, lack. This makes conversion among the three formats relatively simple, and many tools exist that can convert between formats ( Vree et al. 2018 ; Cuthbert & Ariza 2010 ; Cancino-Chacón et al. 2023 ; Pugin et al. 2014 ; Sapp 2012 ) , retaining core musical elements (notes, rests, measures). However, due to differences in the scope of information encoded across formats, certain elements—such as lyrics and articulations—may not be preserved during conversion. We chose ABC as the output format for our model, as it is concise but nearly comprehensive and centers musical (rather than typesetting) elements. We believe the structure of ABC also lends itself well to usage with NLP techniques that help the model learn composite musical concepts, as we will see in the tokenizer (§ 4.1 ). Further, we focus on the recognition of musical notation rather

Read the complete text