Language Model Mapping in Multimodal Music Learning: A Grand Challenge Proposal
Daniel Chin, Gus Xia · 2025
Read Language Model Mapping in Multimodal Music Learning: A Grand Challenge Proposal by Daniel Chin, Gus Xia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00427.txt.
Introduction
Daniel Chin and Gus Xia propose a grand challenge for artificial intelligence: mapping a language model from one domain onto another, in a 2025 paper. Models in fields such as music learn deep patterns, but linking those patterns across domains usually demands huge amounts of paired data. The authors argue that standard alignment works only at the surface, matching tokens or embeddings.
Their challenge asks whether the essence captured by one domain's model can be carried into another's, on the premise that both track the same hidden phenomena. Music is their test domain of choice, since paired data there are scarce. They set a basic form of the task, then a harder form tied to acting on mixed input.
Catalog record
| CREATOR | Daniel Chin, Gus Xia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00427
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language Model Mapping in Multimodal Music Learning: A Grand Challenge Proposal Daniel Chin, Gus Xia Abstract We have seen remarkable success in representation learning and language models (LMs) using deep neural networks. Many studies aim to build the underlying connections among different modalities via the alignment and mappings at the token or embedding level, but so far, most methods are very data-hungry, limiting their performance in domains such as music where paired data are less abundant. We argue that the embedding alignment is only at the surface level of multimodal alignment. In this paper, we propose a grand challenge of language model mapping (LMM), i.e., how to map the essence implied in the LM of one domain to the LM of another domain under the assumption that LMs of different modalities are tracking the same underlying phenomena. We first introduce a basic setup of LMM, highlighting the goal to unveil a deeper aspect of cross-modal alignment as well as to achieve more sample-efficiency learning. We then discuss why music is an ideal domain in which to conduct LMM research. After that, we connect LMM in music with a more general and challenging scientific problem of learning to take actions based on both sensory input and abstract symbols , and in the end, present an advanced version of the challenge problem setup. Introduction Multimodality refers to the use of multiple sensory channels, such as visual, aural, and gestural channels, in processing and communicating information. Different modalities provide diverse perspectives or portrayals of the same objective world. From a time-series perspective, though representations in different modalities may seem to follow distinct rules, such as those prescribed by language models (LMs), those models fundamentally track the moment-to-moment evolution of the same underlying phenomena. This concept forms the basis of what is known as cross-modal alignment . Current machine-learning studies on multimodality are primarily built on the alignment and mapping between multimodal tokens or their embeddings extracted from short segments. Furthermore, the alignments are usually trained by a large quantity of paired data. For example, CLIP [ \citeauthoryearRadford et al.2021 ] adopts contrastive learning to map images and short textual captions to a joint latent space, which serves as a foundation for many follow-up studies on image understanding and generation [ \citeauthoryearKoh, Fried, and Salakhutdinov2024 , \citeauthoryearYu et al.2024 , \citeauthoryearZheng, He, and Wang2023 , \citeauthoryearWu et al.2023b ] . Similarly, CLAP [ \citeauthoryearWu et al.2023c ] learns to align audio clips and text descriptions, and in a recent study, CLaMP [ \citeauthoryearWu et al.2023a ] aligns symbolic music score with text descriptions about the music. (a) (b) Figure 1: (a) The three modalities in music. Vision (V) refers to music score images. Audio (A) refers to music audio. Motion (M) refers to instrument controls, i.e., detailed performance motions. Data in each modality can be modeled by a uni-modal LM. Arrows across modalities refer to time-aligned translation tasks, including OMR and transcription. (b) Illustration of the basic setup. The basic version of our grand challenge involves three given elements and three desired goals. Data are abundantly available in A and V, and the synthesis task is considered to be already well-solved by rule-based music synthesizers as virtual instruments. Given the above, we seek the three goals marked with unique colors: OMR, transcription, and an LM in M. We argue that human learning is obviously more sample-efficient, and the information flow of different modalities connects more deeply in human minds than in current machine-learning models. For example, humans never need to learn from millions of paired video clips and captions to describe videos. Rather, we learn how visual representations of the world flow by observing , and we learn how to write proper texts by reading . These two processes need almost no paired data but they help each other — when reading, the mind imagines the scenes; when watching a movie, the mind narrates what is happening. Based on the cross-modal alignment assumption that these two information flows eventually describe the same underlying world, humans only need a small amount of paired data to calibrate the already developed mapping. The deep connections between different modalities and the resulting sample-efficient benefits are more apparent when we further consider an action model. For example, when learning to improvise on the piano, i.e., to build an action language model, the music we have read and listened to plays a significant role. Musicians learn internal language models for audio and symbolic music, and the musicality implied in the language models can be mapped to the process of piano playing, freeing us from rebuilding the musicality from scratch by literally performing abundant pieces. Additionally, based on the language models, a pianist can learn to sight read a score and to transcribe piano performance from audio with no paired data but purely by trial-and-error on a keyboard. Inspired by the aforementioned observations, we believe the alignment of tokens and embeddings is only at the surface level of multimodal representation learning . In this paper, we propose a grand challenge of Language model mapping (LMM) between different modalities, in order to unveil a deeper aspect of cross-modal alignment and the efficiency of human learning. Music is an ideal domain for LMM research for three reasons. Firstly, music data naturally involve multiple closely aligned modalities that seem superficially distinct: music audio, score, instrument control (MIDI), etc. Secondly, compared to other domains such as robot control and scene rendering, the virtual environment of the music world is concise and clean, as the control of the instruments is low-dimensional and the consequences of actions are simple and locally contained. Lastly, the sparsity of music data compared to vision and text domains forces music AI researchers to develop more sample-efficient solutions. Fig. 1 (a) shows the framework of LMM in music. A, audio, refers to the modality of music audio. V, vision, refers to the modality of score images. M, motion, refers to the modality of instrument controls, i.e., the performance motions. Data within each modality can have their distribution modeled, yielding \mathrm{LM}^{\mathrm{A}} , \mathrm{LM}^{\mathrm{V}} , and \mathrm{LM}^{\mathrm{M}} . Across the modalities are translation tasks including optical music recognition (OMR) and music transcription. The synthesis from H to A is usually considered well-solved by rule-based virtual instruments. The challenge of LMM in music, with a basic example shown in Fig. 1 (b), thus lies in learning LMs or cross-modal maps utilizing not only within-modal data but also data and models from other modalities. Grand Challenge Statement This section defines the grand challenge of Language Model Mapping (LMM) in Multimodal Music Learning by presenting a basic problem setup, contrasting related music AI tasks, and discussing how humans leverage LMM to achieve efficient multimodal learning. Problem Setup: Basic Version The basic version of our grand challenge poses a simple question: Can we learn to play a given instrument by listening to a lot of music and reading a lot of scores? Fig. 1 shows the problem setup. The following elements are considered given : • an abundance of score image data, X^{\mathrm{V}}=\{\mathbf{x}_{i}^{\mathrm{V}}\}_{i=1}^{n}, where \mathbf{x}_{i}^{\mathrm{V}}\in\mathcal{X}^{\mathrm{V}} , • an abundance of music audio data, X^{\mathrm{A}}=\{\mathbf{x}_{i}^{\mathrm{A}}\}_{i=1}^{m}, where \mathbf{x}_{i}^{\mathrm{A}}\in\mathcal{X}^{\mathrm{A}}, • a virtual differentiable piano keyboard capable of synthesizing audio from instrument