Improve Language Model and Brain Alignment via Associative Memory
Congchi YinYongpeng ZhangXuyun Wen, Piji Li · 2025
Read Improve Language Model and Brain Alignment via Associative Memory by Congchi YinYongpeng ZhangXuyun Wen, Piji Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13844.txt.
Introduction
Yin and a team including Zhang, Wen, and Li wrote a twenty twenty-five study on how well large language models match the human brain. People link new speech to past facts held in mind as they listen. The team asks if that same step can help a model track brain waves.
They first map model acts to brain acts for speech. They then add linked facts to the text fed to the model. The match grows in brain zones tied to this kind of recall. Fine-tuned large models match best, aided by a new set of one thousand tales built for the task.
Catalog record
| CREATOR | Congchi YinYongpeng ZhangXuyun Wen, Piji Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13844
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improve Language Model and Brain Alignment via Associative Memory Congchi YinYongpeng ZhangXuyun Wen, Piji Li Abstract Associative memory engages in the integration of relevant information for comprehension in the human cognition system. In this work, we seek to improve alignment between language models and human brain while processing speech information by integrating associative memory. After verifying the alignment between language model and brain by mapping language model activations to brain activity, the original text stimuli expanded with simulated associative memory are regarded as input to computational language models. We find the alignment between language model and brain is improved in brain regions closely related to associative memory processing. We also demonstrate large language models after specific supervised fine-tuning better align with brain response, by building the Association 1 1 1 https://github.com/lemonsis/Association dataset containing 1000 samples of stories, with instructions encouraging associative memory as input and associated content as output. 1 1 footnotetext: Corresponding author. 1 Introduction Human language comprehension is a complicated process widely involving multiple brain functions ( GESCHWIND 1965 ; Aboitiz and Garcıéa V. 1997 ) . Previous studies ( Dronkers et al. 2007 ; Binder 2015 ) have confirmed that Wernicke’s area and Broca’s area are essential in speech comprehension and language production. More relevant regions are found and subdivided to match corresponding functions through functional Magnetic Resonance Imaging (fMRI) scans in later work ( Poremba et al. 2004 ; Gourévitch et al. 2008 ; Chang et al. 2011 ) . Among all the functions related to language comprehension, associative memory ( Anderson and Bower 2014 ) plays an indispensable role, serving as the key to linking together related concepts and pieces of information. Figure 1: An example of how associative memory works when subject listens to speech. The human associative memory system ( Mayes et al. 2007 ; Eichenbaum 2017 ) is responsible for encoding, monitoring, and retrieving diverse components of information, including basic perceptions (i.e. semantic memory ( Binder and Desai 2011 ) ), personal experiences (i.e. episodic memory ( Tulving et al. 1972 ) ), and contextual details (i.e. working memory ( Baddeley 2003 ) ). While associative memory integrates multiple outside stimuli (visual, auditory, sensory, etc.), we primarily focus on associative memory during human language comprehension in this study, particularly in a task involving passively listening to continuous speech. The core effect of associative memory in language comprehension is to help build connections between diverse concepts. Human is capable of retrieving relative concepts that facilitate language understanding instinctively ( Schank 1972 ) . Although associative memory involves complex formation and interaction within the biological brain, at a high level it can be considered as the integration of associated concepts ( McNamara and Magliano 2009 ) . For example, as shown in Figure 1 , the orange box is what associative memory involves. The content in the purple box indicates the speech stimuli that the subject is receiving. The blue box represents previous phonetic stimuli that the subject heard a few moments ago. Large language models like GPT-4 ( Achiam et al. 2023 ) have shown remarkable natural language understanding and generation ability. They follow a next word prediction pattern, which is similar to human’s manner of processing text information ( Caucheteux et al. 2023 ; Antonello and Huth 2024 ) . Previous research ( Jain and Huth 2018 ; Toneva and Wehbe 2019 ; Caucheteux and King 2020 ; Goldstein et al. 2022 ) has confirmed the activations of language models can be linearly mapped to the activity of human brain when receiving the same text stimuli. This finding provides a powerful tool for investigating the alignment between biological brain and language models. For example, Caucheteux et al. 2023 showed such alignment can be improved by introducing future words prediction. Moussa et al. 2024 fine-tuned speech model with brain-relevant semantics to improve its alignment to brain activity. However, as far as we know, few studies explore the associative memory in human language processing. In this paper, we investigate two research questions: (1) Will simulating associative memory in brain language processing improve the alignment between language models and the human brain? (2) Can we improve the alignment between language model and brain by instructing language models to generate associative content? We design the following experiment steps to answer these questions. First, the alignment between language model activation and human brain activity (i.e. brain score) is evaluated when they receive the same text stimuli as input. Following previous studies ( Caucheteux and King 2020 ) , traditional language model GPT-2 ( Radford et al. 2019 ) is selected. We also try large language model LLaMA-2 ( Touvron et al. 2023 ) for comparison. For the first research question, associative memory is considered as the integration of context and associated knowledge, as the example shown in Figure 1 . Data augmentation with simulated associative memory is performed to the original text stimuli. The activation of language model with augmented sentences as input is mapped to the original brain activity. The brain score of regions related to associative memory (e.g. medial temporal lobe (MTL)) is recorded in comparison to the original brain score. For the second research question, we build an instruction tuning dataset Association containing 1000 samples with story paragraphs and instructions encouraging associative memory as input, associated content as output. Association is applied in the supervised fine-tuning (SFT) of base large language model. The brain score of language model after SFT is re-evaluated to see whether the score of relevant regions is improved. Such improvement will become strong evidence suggesting the alignment between large language models and human brain can be improved by instructing language models to generate associative content. Our contributions can be summarized as follows: • We find associative memory simulation via data augmentation is capable of improving language model and brain alignment. • We release Association , an instruction tuning dataset containing 1000 samples for investigating associative memory by supervised fine-tuning large language models. • We demonstrate that fine-tuning a large language model with instructions that promote associative memory can enhance its alignment with brain activity. 2 Related work Language Models and Brain Alignment Previous studies have mapped word-level embeddings to fMRI or MEG signals ( Mitchell et al. 2008 ; Huth et al. 2012 ; Huth et al. 2016 ) . Jain and Huth 2018 ; Toneva and Wehbe 2019 ; Goldstein et al. 2022 ; Caucheteux et al. 2022 indicated that human brain combines information of previous words to predict next words and such prediction is increasingly contextual along the hierarchy by extracting activations from different layers in language models. Such prediction has also been proven to span multiple timescales ( Goldstein et al. 2020 ; Caucheteux et al. 2023 ) . Antonello et al. 2024 further analyzed the mapping of large language models to the human brain. Some studies seek to find the reasons behind the alignment between language models and human brain. Caucheteux et al. 2021 factorized language model activations into lexical, compositional, syntactic and semantic representations. Wehbe et al. 2014 ; OOTA et al. 2024 investigated the specific linguistic properties and brain regions that contribute to such alignment. Figure 2: General framework of calculating brain score, associative