MuseAgent-1: Interactive Grounded Multimodal Understanding of Music Scores and Performance Audio

Qihao Zhao, Yunqi Cao, Yangyu Huang, Hui Yi Leong, Fan Zhang, Kim-Hui Yap, Wei Hu, [0.5em] Nanyang Technological University · 2026

Read MuseAgent-1: Interactive Grounded Multimodal Understanding of Music Scores and Performance Audio by Qihao Zhao, Yunqi Cao, Yangyu Huang, Hui Yi Leong, Fan Zhang, Kim-Hui Yap, Wei Hu, [0.5em] Nanyang Technological University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11968.txt.

Introduction

Qihao Zhao, Yunqi Cao, Yangyu Huang, and their co-authors build a tool that can talk with users about music scores and played audio. Broad multimodal models still grasp music poorly. Music needs grounded thought over both the written score and the sound of a played piece.

Their system, named in the work, joins those two streams. Users can ask about a score and hear links to the audio. Such grounded tools could aid music study and practice. The work tests how well the system ties signs on the page to sound.

Catalog record

CREATORQihao Zhao, Yunqi Cao, Yangyu Huang, Hui Yi Leong, Fan Zhang, Kim-Hui Yap, Wei Hu, [0.5em] Nanyang Technological University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11968

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MuseAgent-1: Interactive Grounded Multimodal Understanding of Music Scores and Performance Audio Qihao Zhao, Yunqi Cao, Yangyu Huang, Hui Yi Leong, Fan Zhang, Kim-Hui Yap, Wei Hu, [0.5em] Nanyang Technological University Abstract Despite recent advances in multimodal large language models (MLLMs), their ability to understand and interact with music remains limited. Music understanding requires grounded reasoning over symbolic scores and expressive performance audio, which general-purpose MLLMs often fail to handle due to insufficient perceptual grounding. We introduce MuseAgent, a music-centric multimodal agent that augments language models with structured symbolic representations derived from sheet music images and performance audio. By integrating optical music recognition and automatic music transcription modules, MuseAgent enables multi-step reasoning and interaction over fine-grained musical content. To systematically evaluate music understanding capabilities, we further propose MuseBench, a benchmark covering music theory reasoning, score interpretation, and performance-level analysis across text, image, and audio modalities. Experiments show that existing MLLMs perform poorly on these tasks, while MuseAgent achieves substantial improvements, highlighting the importance of structured multimodal grounding for interactive music understanding. 1 Introduction Figure 1: Overview of MuseBench and MuseAgent . MuseBench consists of multimodal music understanding tasks across text, image, and audio modalities, covering music theory, sheet score analysis, performance interpretation. MuseAgent integrates these modalities via sheet symbolic recognition, audio alignment, and retrieval modules, enabling large language models to answer complex music questions. “O Muses, O high genius, now help me!” — Dante, Inferno , Canto II, line 7 Music is a structured yet expressive domain, making it a compelling testbed for artificial intelligence. It spans symbolic representations such as notated scores and expressive acoustic realizations such as performances, each requiring distinct perceptual and reasoning capabilities. As both a formal system and an emotional medium, music challenges AI models to reason across modalities with high precision and contextual nuance 9 ; 7 . These requirements expose fundamental limitations of current multimodal systems, particularly large language models that lack structured grounding in symbolic and temporal representations. Among musical forms, piano music occupies a uniquely central role in the Western repertoire, supported by centuries of standardized notation and extensive performance archives. Its wide pitch range, dense polyphony, and two-hand coordination make piano music a particularly demanding stress test for multimodal reasoning systems 14 . Understanding piano music requires jointly modeling complex symbolic structures—such as pitch, rhythm, and dynamics—and expressive performance attributes including timing, articulation, and rubato. This integration is essential for interactive applications such as transcription, accompaniment, digital archiving, and music education 4 . Despite significant progress in isolated tasks, existing approaches remain insufficient for holistic and interactive music understanding. Prior work has advanced single-modality problems such as Optical Music Recognition (OMR) for scores and Automatic Music Transcription (AMT) for audio. Large-scale datasets like MAESTRO 14 and generative models such as MusicLM 2 demonstrate the feasibility of modeling symbolic and acoustic music representations, yet they fall short of enabling fine-grained cross-modal reasoning. Recent Multimodal Large Language Models (MLLMs), including GPT-4o 30 and Gemini 10 , promise general cross-modal capabilities but perform poorly on music understanding tasks that require precise symbolic parsing or long-form performance analysis. This failure stems largely from the lack of domain-specific perceptual grounding and the inability to align symbolic and acoustic modalities at high temporal resolution. Several recent systems have attempted to bridge this gap by augmenting LLMs with external tools. AudioGPT 17 and MusicAgent 44 represent early efforts toward music-aware agents by coupling language models with domain-specific components. However, these systems often struggle with complex notation, long-duration performance recordings, and structured interaction, and they lack systematic evaluation protocols tailored to music understanding. Meanwhile, advances in Retrieval-Augmented Generation (RAG) 19 ; 12 ; 5 demonstrate that grounding language models in structured external representations can substantially improve domain-specific reasoning and reduce hallucination. Existing multimodal RAG frameworks 35 ; 27 ; 26 , however, primarily focus on text–vision–speech settings and rarely address music, where symbolic scores and expressive audio must be tightly aligned. To address these challenges, we propose MuseAgent, a multimodal retrieval-augmented agent designed specifically for music understanding and interaction. MuseAgent integrates a large language model with specialized perceptual front-ends: (i) a measure-wise OMR module that converts sheet music images into symbolic representations such as ABC notation 47 , (ii) an AMT-based performance analysis module that aligns audio recordings with MusicXML scores and extracts expressive features in structured JSON formats, and (iii) a retrieval module that enables both explicit and implicit access to symbolic and audio libraries. These components ground the language model in structured multimodal representations, while a memory bank supports long-context, multi-turn reasoning over musical content. Evaluating such music understanding agents poses a non-trivial challenge. Existing datasets and benchmarks focus primarily on transcription or generation, and do not assess an agent’s ability to reason interactively across symbolic scores and performance audio. To enable systematic evaluation, we introduce MuseBench, the first benchmark designed to assess multimodal music understanding agents. Centered on piano repertoire, MuseBench includes tasks such as score–audio alignment, performance error detection, and expressive deviation analysis, repurposing resources like MAESTRO into high-level reasoning tasks suitable for evaluating agentic music understanding. Our experiments on MuseBench reveal that general-purpose MLLMs exhibit limited capabilities in fine-grained symbolic and performance-level music reasoning. In contrast, MuseAgent demonstrates substantial improvements on challenging image- and audio-based tasks, validating the effectiveness of structured perceptual grounding and agent-based multimodal reasoning for interactive music understanding. 2 Related Work Music Understanding Systems and Agents. Recent advances in general-purpose agents, such as ReAct, Auto-GPT, and Gorilla, have demonstrated the effectiveness of tool-augmented reasoning for complex tasks. In the music domain, early systems such as MuseNet 33 and MusicLM 2 primarily focus on symbolic or audio music generation, rather than music understanding or interactive reasoning. More recent music-oriented multimodal language models adopt domain-specific training strategies. For example, MusiLingo 8 targets music captioning and question answering via instruction-tuned audio–language modeling, MuMu-LLaMA 25 unifies music audio, images, and language within a single multimodal framework, and NotaGPT 37 focuses on music notation understanding through a vision–language model. SymphonyNet 24 further demonstrates symbolic music generation at orchestral scale. While these domain-specific models highlight the value of multimodal alignment and specialized training, they are typically designed as monolithic architectures and do not explicitly support structured interaction, tool invocation, or multi-step reasoning. In parallel, agent-based systems such as MusicAgent 44 and AudioGPT 17 begin to couple large language models with domain-specific tools for processing music inputs. Although promising, these systems are not designed for fine-grained music understanding that requires jointly reasoning over symbolic scores and expressive performance audio, nor do they target complex piano repertoire. In contrast, our proposed MuseAgent advances this line of work by introducing a music-centric agent architecture that integrates structured perceptual grounding, retrieval-based reasoning, and agentic orchestration across symbolic and acoustic modalities. Music Understanding Benchmarks. A critical challenge in developing music understanding agents lies in evaluation. Existing music datasets, such as MAESTRO 15 and URMP 20 , provide aligned score–audio pairs and have been widely used for transcription, alignment, and generation tasks. However, these resources lack task-oriented evaluation protocols that assess high-level reasoning or interactive understanding. Recent multimodal question-answering datasets, including MUSIC-AVQA 21 and MuChoMusic 42 , extend music benchmarks toward Q&A settings, but do not emphasize detailed symbolic score interpretation or nuanced

Read the complete text