MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation
Daeyong Kwon, SeungHeon Doh, Juhan Nam · 2025
Read MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation by Daeyong Kwon, SeungHeon Doh, Juhan Nam (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23334.txt.
Introduction
Daeyong Kwon, SeungHeon Doh, and Juhan Nam give language models a music library so they can answer music questions with facts, not flair. Their paper from twenty twenty-five notes that music forms a thin slice of model training data. Zero-shot skill on music text tasks thus lags. Their MusT-RAG frame fetches outside knowledge at answer time.
A music-tuned vector store, MusWikiDB, backs the retrieval stage. Fetched context is used both when fine-tuning and when answering live. That pairing turns a general model into a music hand. Tests on in-domain and out-of-domain music sets beat plain fine-tuning across the board. The custom store also beat general Wikipedia text on score and speed. The work shows retrieval can carry a niche domain that training data slighted.
Catalog record
| CREATOR | Daeyong Kwon, SeungHeon Doh, Juhan Nam |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-23334
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation Daeyong Kwon, SeungHeon Doh, Juhan Nam Abstract Recent advancements in Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains. While they exhibit strong zero-shot performance on various tasks, LLMs’ effectiveness in music-related applications remains limited due to the relatively small proportion of music-specific knowledge in their training data. To address this limitation, we propose MusT-RAG , a comprehensive framework based on Retrieval Augmented Generation (RAG) to adapt general-purpose LLMs for text-only music question answering (MQA) tasks. RAG is a technique that provides external knowledge to LLMs by retrieving relevant context information when generating answers to questions. To optimize RAG for the music domain, we (1) propose MusWikiDB, a music-specialized vector database for the retrieval stage, and (2) utilizes context information during both inference and fine-tuning processes to effectively transform general-purpose LLMs into music-specific models. Our experiment demonstrates that MusT-RAG significantly outperforms traditional fine-tuning approaches in enhancing LLMs’ music domain adaptation capabilities, showing consistent improvements across both in-domain and out-of-domain MQA benchmarks. Additionally, our MusWikiDB proves substantially more effective than general Wikipedia corpora, delivering superior performance and computational efficiency. 1 Introduction Figure 1 : Overview of our MusT-RAG framework. The retriever searches for relevant information in MusWikiDB based on similarity for music-related queries, and augments the generator’s prompt with this information to generate an answer. Recent advancements in Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks, thanks to their massive scale and ability to generalize across diverse domains. However, LLMs still face significant limitations in music-related applications due to the relatively small amount of music-specific knowledge in their training data. To effectively deploy general LLMs in music-related domains such as music recommendation systems and chatbots, a deep understanding of Music Question Answering (MQA) in text-only settings is essential. Mastering text-based MQA would enable LLMs to provide more accurate and contextually aware responses to user questions about music, ultimately enhancing the user experience in music-related applications. Developing a robust text-only music QA framework is therefore a key step toward improving the adaptability of LLMs in the music domain. Traditionally, domain adaptation of LLMs has often been achieved by fine-tuning them on domain-specific data [ 1 , 2 , 3 ] . However, this approach faces challenges in securing high-quality training data, and as model size increases, the training time and cost also rise significantly. Additionally, continuously updating the model with new knowledge remains a persistent challenge. In this paper, we propose MusT-RAG , a framework that leverages Retrieval Augmented Generation (RAG) [ 4 ] techniques to enhance general-purpose LLMs for music-specific tasks. The core idea behind MusT-RAG is to augment LLMs with external knowledge retrieval mechanisms. Specifically, the model retrieves relevant external knowledge from a pre-constructed, comprehensive music-specific vector database in order to answer input questions. For music-domain specific retrieval, we introduce MusWikiDB , which, to our knowledge, is the first comprehensively curated vector database designed specifically for music-related content. We explore various design choices for optimizing retrieval performance, including embedding models and chunking strategies. By incorporating this retrieval process, MusT-RAG enables LLMs to efficiently generate contextually relevant responses, drawing on specialized music knowledge to enhance performance on music-related tasks, all without requiring additional training. Furthermore, we extend the application of RAG beyond inference by incorporating contextual information during the fine-tuning process. Our empirical analysis reveals that this context-aware training enhances the model’s contextual understanding capabilities, defined as the ability to generate coherent and relevant text within a specific context [ 5 ] , outperforming conventional fine-tuning approaches. MusT-RAG demonstrated the effectiveness of RAG across all scenarios including in-domain and out-of-domain settings, as well as both fine-tuning and inference stages. By retrieving relevant context information from the database, MusT-RAG effectively addresses the music domain adaptation problem. Our contributions are as follows: i) We propose the MusT-RAG framework, which leverages RAG to retrieve relevant context from a music-specific database for answer generation. ii) We create MusWikiDB , the first comprehensive music-specific vector database for RAG. iii) We demonstrate that RAG-style fine-tuning can resolve the issue of decreased contextual understanding performance with conventional fine-tuning. iv) We introduce ArtistMus , a benchmark designed to evaluate artist-related questions in text-only MQA tasks, addressing a gap in existing evaluations. 2 Music Question Answering Question Answering (QA) refers to the task of providing an appropriate answer to a given question, which is one of the Information Retrieval (IR) tasks [ 6 ] . The task is typically framed as retrieving relevant information from a collection of documents or knowledge sources to answer fact-based questions. Open-domain QA involves answering questions from a vast and varied set of topics using a large collection of general knowledge documents [ 7 , 8 ] . In contrast, domain-specific QA targets specialized fields such as medicine [ 9 ] , law [ 10 ] , or music [ 11 , 12 , 13 , 14 ] , where both the document set and the questions are confined to that domain. In this work, we define the MQA task as the problem of providing accurate and relevant answers to music-related questions by leveraging domain-specific musical knowledge. Several recent studies have introduced music-related benchmarks to evaluate LLM performance. MuChoMusic [ 11 ] features 1,187 audio-based multiple-choice questions that assess both musical knowledge and reasoning capabilities. MusicTheoryBench [ 12 ] contains 372 expert-validated questions designed to evaluate advanced music knowledge and reasoning skills. TrustMus [ 13 ] comprises 400 questions across four domains— People , Instruments and Technology , Genres, Forms, and Theory , and Culture and History —all derived from The Grove Dictionary Online [ 15 ] . ZIQI-Eval [ 14 ] presents a comprehensive evaluation framework consisting of 14,000 comprehension tasks that span 10 major topics and 56 subtopics, encompassing a broad spectrum of music-related knowledge. A significant shortcoming of existing benchmarks is their inadequate representation of rich metadata about tracks, artists, and albums—information crucial for everyday music listening contexts. Current text-only QA benchmarks inadequately address common music information needs, lacking comprehensive coverage of details that listeners frequently seek: complete discographies, artist collaboration networks, creative evolution across albums, and notable career achievements. This pronounced disparity between current MQA capabilities and the practical information demands of music consumers highlights the pressing need for benchmarks specifically designed to evaluate responses to artist-centric quetions. 3 Retrieval Augmented Generation 3.1 RAG Framework Retrieval-Augmented Generation (RAG) enhances the capabilities of LLMs by combining their generative abilities with access to external knowledge. Instead of relying solely on parametric memory, RAG retrieves relevant passages from an external database during inference time to ground responses in factual context. 3.1.1 Indexing The first step in RAG is constructing a searchable knowledge database. This involves segmenting a large corpus into fixed-size text passages (chunking), followed by representing each passage using an embedding models. Various embedding models can be used for indexing: Sparse Embeddings [ 16 , 17 ] use term frequency-based scoring to match exact keywords, offering fast and interpretable retrieval for large-scale datasets. Dense Embeddings [ 18 , 19 , 17 ] map questions and documents into a shared vector space, enabling semantic matching beyond keyword overlap. Audio-Text Joint Embeddings [ 20 , 21 , 22