Predicting Artificial Neural Network Representations to Learn Recognition Model for Music Identification from Brain Recordings
Taketo Akama, Zhuohao Zhang, Pengcheng Li, Kotaro Hongo, Hiroaki Kitano, Shun Minamikawa, Natalia Polouliakh · 2024
Read Predicting Artificial Neural Network Representations to Learn Recognition Model for Music Identification from Brain Recordings by Taketo Akama, Zhuohao Zhang, Pengcheng Li, Kotaro Hongo, Hiroaki Kitano, Shun Minamikawa, Natalia Polouliakh (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15560.txt.
Introduction
Taketo Akama and six co-authors published a paper in 2024 on teaching a machine to recognize music from brain waves by aiming at an artificial network's inner picture of the song. Brains and networks form strikingly similar representations of the same sounds, and earlier work predicted brain activity from network features. This team reversed the arrow: network representations became the teacher.
Volunteers' scalp recordings while listening to music trained a model whose target was not the song's name but the representation a music-identifying network builds for it. Classification accuracy jumped substantially over direct training. The trick turns a strong artificial listener into a scaffold for decoding a noisy human one, with implications for brain-computer interfaces and the science of music perception.
Catalog record
| CREATOR | Taketo Akama, Zhuohao Zhang, Pengcheng Li, Kotaro Hongo, Hiroaki Kitano, Shun Minamikawa, Natalia Polouliakh |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15560
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Predicting Artificial Neural Network Representations to Learn Recognition Model for Music Identification from Brain Recordings Taketo Akama, Zhuohao Zhang, Pengcheng Li, Kotaro Hongo, Hiroaki Kitano, Shun Minamikawa, Natalia Polouliakh Abstract Recent studies have demonstrated that the representations of artificial neural networks (ANNs) can exhibit notable similarities to cortical representations when subjected to identical auditory sensory inputs. In these studies, the ability to predict cortical representations is probed by regressing from ANN representations to cortical representations. Building upon this concept, our approach reverses the direction of prediction: we utilize ANN representations as a supervisory signal to train recognition models using noisy brain recordings obtained through non-invasive measurements. Specifically, we focus on constructing a recognition model for music identification, where electroencephalography (EEG) brain recordings collected during music listening serve as input. By training an EEG recognition model to predict ANN representations—representations associated with music identification—we observed a substantial improvement in classification accuracy. This study introduces a novel approach to developing recognition models for brain recordings in response to external auditory stimuli. It holds promise for advancing brain-computer interfaces (BCI), neural decoding techniques, and our understanding of music cognition. Furthermore, it provides new insights into the relationship between auditory brain activity and ANN representations. Introduction Neural decoding in the audio domain refers to the technology for extracting or reconstructing audio stimuli or their features from brain recordings obtained when the audio stimulus is processed by the brain 1 , 2 , 3 , 4 , 5 , 6 . This technique provides insights into the neural localization and dynamics relevant to auditory stimuli and their features 7 , 5 , 8 . Furthermore, it enables the exploration of individual differences in cognition, such as those between experts and non-experts or native and non-native listeners 8 , 9 . Neural decoding in the audio domain also has various research and industrial applications as part of brain-computer interface (BCI) systems, including speech production 2 , 3 , 4 , 10 , music identification 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 6 , and music reconstruction 8 , 1 , 5 , 21 . Methods for measuring brain responses to auditory stimuli include invasive techniques such as electrocorticography (ECoG) and non-invasive approaches such as functional magnetic resonance imaging (fMRI), electroencephalography (EEG), magnetoencephalography (MEG), and near-infrared spectroscopy (NIRS) 22 . Among these, EEG is particularly suited for high-temporal-resolution time-series data due to its high sampling rate 23 , 22 . Its affordability, ease of use, and portability suggest significant potential for widespread adoption 22 . However, EEG faces persistent challenges, including high noise levels and signal attenuation from subcortical regions 23 , 24 . Recently, studies have reported that artificial neural network (ANN) representations exhibit similarities to cortical representations in response to the same auditory sensory input, i.e., the same auditory stimulus 9 , 25 , 26 , 27 . These studies explore the feasibility of predicting cortical representations by regressing from ANN representations to cortical representations, thereby probing the alignment between these two types of representations. Our hypothesis posits that if auditory brain response recordings under ideal conditions resemble the representations of artificial neural networks (ANNs), then the information contained in ANN representations may be useful for complementing incomplete brain response recordings obtained under less-than-ideal conditions. We attempt to validate this hypothesis by treating ANN representations as the target signals to be predicted, training a transformation function that complements the measured brain response information, and observing improvements in the accuracy of neural decoding models (recognition models) utilizing the complemented brain response information. In prior studies evaluating the similarity between brain and ANN representations in response to auditory stimuli, cortical representations were predicted from ANN representations using small linear models, and a high prediction accuracy was interpreted as evidence of similarity. In contrast, our approach reverses this direction of prediction. Specifically, we train a transformation model to enable recorded auditory brain responses to predict ANN representations. Rather than merely evaluating similarity, our objective is to leverage the similarity for information complementation. For this reason, we employ larger, non-linear models. The ultimate goal is to effectively train auditory neural decoding models (recognition models) even when noisy brain recordings are obtained through non-invasive measurements, by using ANN representations as the target signals to be predicted. Figure 1 illustrates the conceptual framework of our approach. Figure 1 : Conceptual framework of our approach: predicting ANN representation to learn auditory EEG recognition model. When constructing a recognition model that uses brain recordings obtained in response to an auditory stimulus as input, the model is trained to predict the ANN representation obtained by inputting the same auditory stimulus into the ANN. This framework improves the performance of the recognition model by effectively utilizing the findings that cortical representations and ANN representations resemble each other when the same auditory stimulus is inputted. In this study, we focus on developing a recognition model for music identification using EEG brain recordings collected during music listening. By training a recognition model with EEG inputs to infer representations capable of predicting the ANN representations used for music identification, we observed substantial improvements in classification accuracy. Furthermore, the model demonstrated robust learning that was less dependent on the initialization of model weights. The most significant improvement in classification accuracy was observed when assuming a brain response delay of approximately 200 ms to music stimuli. This aligns with recent scientific findings on brain response delays to music stimuli 8 , suggesting that the model effectively utilizes the similarity between ANN and brain representations. Additionally, the model exhibited an ideal property where longer durations of EEG input resulted in higher music identification accuracy. We also report on the variations in accuracy based on individual differences and differences in the music itself, offering insights from a neuroscience perspective. To predict ANN representations, various methods can be considered. In this work, we propose using a contrastive learning approach 28 , 29 , 30 . Unlike conventional contrastive learning, our method incorporates two major techniques. The first technique involves simultaneously solving a classification task on both the brain representation branch and the ANN representation branch when using ANN representations as the supervisory signal. This dual-task setup reduces the learning of irrelevant features, generates supervisory signals specific to the target task, and focuses the model on learning features essential for the current task. The second technique is that the ANN representation branch is not trained to align with the brain representation branch. Aligning the ANN representation branch too closely with the brain branch risks diminishing the discriminative capability of the ANN representations for the target task due to the noise inherent in brain recordings. In practice, we confirmed through experiments that implementing this second technique improves music identification accuracy. This study proposes a novel approach to recognition models for brain recordings in response to music auditory stimuli. It holds promise for advancing the understanding of cognitive mechanisms through neural decoding, contributing to the development of brain-computer interfaces (BCI), and offering valuable insights into the relationship between the human music perception and ANN processing of music. Results This study utilizes the NMED-T dataset, comprising EEG data collected from 20 subjects as they