Musi ๐• \mathbb{X} QA: Advancing Visual Music Understanding in Multimodal Large Language Models

Jian Chen, Wenye Ma, Penghang Liu, Wei Wang, Tengwei Song, Ming Li, Chenguang Wang, Ruiyi Zhang, Changyou Chen, [0.5em] University at Buffalo ยท 2025

Read Musi ๐• \mathbb{X} QA: Advancing Visual Music Understanding in Multimodal Large Language Models by Jian Chen, Wenye Ma, Penghang Liu, Wei Wang, Tengwei Song, Ming Li, Chenguang Wang, Ruiyi Zhang, Changyou Chen, [0.5em] University at Buffalo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23009.txt.

Introduction

Jian Chen and co-authors build the first broad test set for music sheet reading by AI in a 2025 paper. Multimodal models read photos, docs, and charts with skill, yet sheet music sits near blind. MusiXQA fills the gap with fake sheets drawn by a music type tool. Notes on pitch, beat length, chords, clefs, and time marks come with each sheet. Tasks span that full range in visual question form.

Tests show top models stumble hard on sheet sense. Their scores fall far short of their doc scores. The team then tuned a model on the set, named Phi-3-MusiX. It beat the big GPT-based tools by a clear span. The set and model are due for free release. The authors frame sheet sense as a fresh test of true visual thought.

Catalog record

CREATORJian Chen, Wenye Ma, Penghang Liu, Wei Wang, Tengwei Song, Ming Li, Chenguang Wang, Ruiyi Zhang, Changyou Chen, [0.5em] University at Buffalo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-23009

Browse the full library ยท About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Musi ๐• \mathbb{X} QA: Advancing Visual Music Understanding in Multimodal Large Language Models Jian Chen, Wenye Ma, Penghang Liu, Wei Wang, Tengwei Song, Ming Li, Chenguang Wang, Ruiyi Zhang, Changyou Chen, [0.5em] University at Buffalo Abstract Multimodal Large Language Models (MLLMs) have achieved remarkable visual reasoning abilities in natural images, text-rich documents, and graphic designs. However, their ability to interpret music sheets remains underexplored. To bridge this gap, we introduce Musi ๐• \mathbb{X} QA 1 , the first comprehensive dataset for evaluating and advancing MLLMs in music sheet understanding. Musi ๐• \mathbb{X} QA features high-quality synthetic music sheets generated via MusiX T e X , with structured annotations covering note pitch and duration, chords, clefs, key/time signatures, and text, enabling diverse visual QA tasks. Through extensive evaluations, we reveal significant limitations of current state-of-the-art MLLMs in this domain. Beyond benchmarking, we developed Phi-3-MusiX, an MLLM fine-tuned on our dataset, achieving significant performance gains over GPT-based methods. The proposed dataset and model establish a foundation for future advances in MLLMs for music sheet understanding. Code, data, and model will be released upon acceptance. \@LN@col 1 \@LN@col 2 \@LN@col 1 \@LN@col 2 \@LN@col 1 \@LN@col 2 \@LN@col 1 \@LN@col 2 \@LN@col 1 \@LN@col 2 \@LN@col 1 \@LN@col 2 \@LN@col 1 \@LN@col 2 \@LN@col 1 \@LN@col 2 \@LN@col 1 \@LN@col 2 \@LN@col 1 \@LN@col 2 \@LN@col 1 \@LN@col 2 โ€  โ€  footnotetext: 1 Code is available at https://github.com/puar-playground/MusiXQA 1 Introduction Multimodal Large Language Models (MLLMs) are becoming general-purpose reading assistants for visual content, capable of interpreting natural images and text-rich documents [ 1 ] . However, existing models still struggle with visual question answering tasks involving music sheets, performing at near-random levels [ 2 ] , indicating that this modality remains underexplored. Enabling MLLMs to read and reason over music sheets would be a valuable extension of their capabilities, as sheet music plays a central role in how music is taught, analyzed, and communicated. Although music notation is a symbolic and lossy approximation of sound, music sheets remain the only widely accepted visual system for the written transmission of music. It serves as a crucial bridge between audio, visual, and textual representations, making it an essential modality for large language model-based AI systems to effectively interpret and understand music. Unlike tasks such as reading visual text or recognizing objects in images, where answers are often apparent to the human eye, reading music notation requires interpreting dense symbolic structures [ 3 ] . Even for humans, achieving proficiency in reading music typically requires years of dedicated training [ 4 ] . As a result, music sheet understanding poses a uniquely complex challenge, where AI assistance is not only beneficial to human but arguably more necessary than in many other visual understanding tasks. Figure 1 : Data generation and model workflow in Musi \mathbb{X} QA. Music metadata is sampled and rendered into sheet images via MusiX T e X , with QA pairs generated from templates. The resulting data is used to train and evaluate MLLMs on visual music understanding tasks. The traditional approach to this task is Optical Music Recognition (OMR) [ 5 , 6 ] , a field with a long research history [ 7 , 8 , 9 ] . Many existing OMR systems still rely on relatively small neural networks and multi-stage pipelines tailored to specific sub-tasks. In contrast, recent advances in MLLMs have shown strong performance across a range of visual tasks [ 10 , 11 , 12 , 13 , 14 ] , demonstrating that modern architectures can perform end-to-end reasoning over structured visual inputs directly in natural language format, without relying on traditional detection modules such as Optical Character Recognition (OCR) tools [ 15 , 16 ] . These developments suggest that MLLMs offer a promising alternative to pipeline-based approaches in domains like music sheet understanding, where such capabilities remain underexplored. To bridge this gap, we introduce Musi \mathbb{X} QA, a large-scale benchmark dataset designed to evaluate and enhance MLLMs for music sheet understanding. Figure 1 provides an overview of our data generation and model inference workflow. Musi \mathbb{X} QA contains 9,600 high-quality synthetic music sheets rendered via MusiX T e X , paired with over 130,000 visual question-answer (QA) pairs spanning OCR, layout understanding, optical music recognition, and chord estimation. Unlike prior datasets, Musi \mathbb{X} QA offers diverse and balanced coverage of musical concepts, enabling fine-grained evaluation across multiple tasks. In addition, we develop Phi-3-MusiX, a fine-tuned version of Phi-3-Vision, adapted using parameter-efficient LoRA training on the Musi \mathbb{X} QA dataset. Experimental results show that existing MLLMs, including GPT-4o and Paligemma2, struggle with music sheets reading, while Phi-3-MusiX achieves up to eight times performance gains in GPT evaluation accuracy on the OMR-based task. These results highlight the importance of domain-specific supervision and the value of Musi \mathbb{X} QA as a resource to enable visual music understanding in MLLMs. Our contributions are as follows: โ€ข We propose Musi \mathbb{X} QA, the first large-scale, diverse, and balanced synthetic dataset for visual question answering on music sheets. โ€ข We develop Phi-3-MusiX, the first MLLM fine-tuned for music sheet understanding, which significantly outperforms the best baseline on our benchmark, demonstrating the effectiveness of our dataset in advancing symbolic music reasoning in MLLMs. โ€ข We introduce a MusiX T e X based framework for scalable generation of music sheets, supporting both random/controlled data sampling and conversion from real-world MIDI data. โ€ข We propose kern+ , a compact symbolic representation designed for efficient modeling of pitch and duration, and analyze how output format influences training dynamics and model performance. 2 Related Work 2.1 Music Sheet Benchmarks and Datasets The development of OMR and music sheet understanding has been supported by various datasets, each addressing different aspects of the task. CVC-MUSCIMA [ 17 ] and MUSCIMA++ [ 18 ] focus on handwritten music scores, tackling challenges in staff line removal and symbol segmentation. DeepScores [ 19 ] shifts toward typeset music, enabling fine-grained OMR tasks. For symbolic recognition, PrIMuS [ 20 ] introduces monophonic scores with semantic and agnostic representations, while Camera-PrIMuS simulates real-world distortions in sheet music capture. DoReMi [ 21 ] further explores single-line typeset music with note variations but omits key and time signature annotations. More recent efforts aim at structured evaluation. COMREF [ 22 ] provides over 400k measure-level typeset scores, using Music Tree Notation (MTN) for a standardized output representation. OLiMPiC [ 23 ] offers scanned and synthetic system-level sheets, while MMMU [ 2 ] evaluates MLLMs via QA over system-level music scores, revealing performance close to random guessing. Despite these advances, existing datasets often exhibit distributional bias due to their reliance on real-world sources. In contrast, our dataset is synthetically generated with controlled diversity across musical attributes such as key, note density, and measure length. It provides a large-scale, full-page QA benchmark tailored for MLLMs, supporting comprehensive evaluation across multiple tasks. 2.2 Optical Music Recognition OMR converts sheet music images into digital formats such as MIDI or MusicXML, aiming to capture the complexity and variability of musical notation. Traditional OMR systems often rely on modular pipelines that separately handle staff line detection, symbol classification, and structural analysis. For example, Oemer [ 24 ] employs two UNet models alongside

Read the complete text