Distinct social-linguistic processing between humans and large audio-language models: Evidence from model-brain alignment
Hanlin Wu, Xufeng Duan, Zhenguang G. Cai · 2025
Read Distinct social-linguistic processing between humans and large audio-language models: Evidence from model-brain alignment by Hanlin Wu, Xufeng Duan, Zhenguang G. Cai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19586.txt.
Introduction
Hanlin Wu and co-authors published this comparison study in 2025. Hearing a voice tells us who is speaking, and listeners fold that knowledge into understanding words. The team asked whether large audio-language models do anything similar. They compared two such models with human brain recordings. Their tests used sentences that clash with the speaker, either against social stereotypes or against basic biology. A man saying he gets manicures was one social example; a man claiming pregnancy was a biological one.
One model, Qwen2-Audio, showed surprise at speaker clashes, and its surprise scores predicted a known human brain response. The second model, Ultravox 0.5, barely reacted to speaker traits. Neither model copied the human split between the two clash types, which trigger different brain signals in people. The authors conclude that current models partly track speaker context but process it through different mechanisms than the human brain.
Catalog record
| CREATOR | Hanlin Wu, Xufeng Duan, Zhenguang G. Cai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19586
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distinct social-linguistic processing between humans and large audio-language models: Evidence from model-brain alignment Hanlin Wu, Xufeng Duan, Zhenguang G. Cai Abstract Voice-based AI development faces unique challenges in processing both linguistic and paralinguistic information. This study compares how large audio-language models (LALMs) and humans integrate speaker characteristics during speech comprehension, asking whether LALMs process speaker-contextualized language in ways that parallel human cognitive mechanisms. We compared two LALMs’ (Qwen2-Audio and Ultravox 0.5) processing patterns with human EEG responses. Using surprisal and entropy metrics from the models, we analyzed their sensitivity to speaker-content incongruency across social stereotype violations (e.g., a man claiming to regularly get manicures) and biological knowledge violations (e.g., a man claiming to be pregnant). Results revealed that Qwen2-Audio exhibited increased surprisal for speaker-incongruent content and its surprisal values significantly predicted human N400 responses, while Ultravox 0.5 showed limited sensitivity to speaker characteristics. Importantly, neither model replicated the human-like processing distinction between social violations (eliciting N400 effects) and biological violations (eliciting P600 effects). These findings reveal both the potential and limitations of current LALMs in processing speaker-contextualized language, and suggest differences in social-linguistic processing mechanisms between humans and LALMs. 1 Introduction Humans are remarkably adept at extracting speaker characteristics from vocal cues. Within milliseconds of hearing a voice, listeners can perceive a speaker’s gender, age, health condition, personality traits, and other socio-demographic attributes ( Lavan et al. 2024 ) . The perceived speaker attributes then form a critical context for language comprehension, shaping how linguistic input is processed and interpreted ( Wu and Cai 2024a ) . For example, when we hear someone say “The first time I got pregnant I had a hard time,” it is straightforward when coming from a female speaker but would be puzzling if a man were to say it. Electroencephalography (EEG) studies show that when people hear sentences containing speaker incongruencies—such as “The first time I got pregnant …” spoken by a man (violating biological knowledge) or “I like to get manicures …” spoken by a man (violating gender stereotypes)—their brain responses diverge from speaker-congruent conditions, showing an N400 effect ( Martin et al. 2016 ; Van Berkum et al. 2008 ; Van den Brink et al. 2012 ) or a P600 effect ( Lattner and Friederici 2003 ; Foucart et al. 2015 ) . These neural responses show that speaker characteristics actively shape the real-time processing of spoken language. The human capacity for speaker-contextualized language processing has recently been explained through a rational inference framework ( Wu and Cai 2024b ) . This framework proposes that humans engage in rational inference during real-time language comprehension—a process where listeners actively reason about the most likely interpretation given both linguistic input and speaker characteristics. Using social-stereotype violation (e.g., men getting manicures) and biological-knowledge violation (e.g., men getting pregnant) as test cases, they showed that when encountering speaker-content mismatches that violate social stereotypes, listeners can still arrive at a “literal” interpretation through effortful integration with their social knowledge, reflected in N400 effects. However, when faced with biological impossibilities, listeners rationally infer potential errors in the input and engage in error correction processes, manifested as P600 effects. Recent advances in large language models (LLMs) have demonstrated increasing capabilities in contextual understanding ( Zhu et al. 2024 ) and multimodal processing ( Wang et al. 2024 ; Zhang et al. 2024 ) . While initially focused on text, these models have expanded into multimodal tasks, showing remarkable abilities in integrating inputs from diverse modalities like vision and speech. This evolution has led to the development of large audio-language models (LALMs) that can process audio inputs, including speaker characteristics, acoustic features, along with other contextual information. The integration of LLMs into audio processing has progressed through several stages ( Peng et al. 2024 ) . Early attempts focused on incorporating Transformer architectures into traditional speech models, as exemplified by HuBERT’s self-supervised learning on unlabeled speech data ( Hsu et al. 2021 ) . More recent approaches have shifted toward direct audio processing with LLMs by mapping audio features to tokens, not only for higher computational efficiency but also enabling richer paralinguistic processing through end-to-end multimodal integration (e.g., Chu et al. 2024 ). This paradigm shift has produced models that are capable of increasingly complex tasks: AudioPaLM can preserve speaker voice characteristics during speech processing and generation ( Rubenstein et al. 2023 ) , SALMONN can perform audio-based storytelling and speech-audio co-reasoning ( Tang et al. 2023 ) , and Qwen2-Audio can explicitly identify speaker demographics and emotions ( Chu et al. 2024 ) . These emerging abilities raise questions about whether LALMs process speaker-contextualized language in ways that parallel human cognitive mechanisms. As these models are increasingly deployed in interactive settings where they must interpret and respond to diverse speakers, understanding their social-linguistic processing has both theoretical and practical implications. On the one hand, comparing LALMs with human processing can provide insights into models’ emergent cognitive mechanisms, an approach that has been widely used with deep neural networks ( AlKhamissi et al. 2024 ; AlKhamissi et al. 2025 ; Schrimpf et al. 2018 ) ; on the other hand, identifying divergences between human and model processing helps pinpoint potential limitations in current architectures or training method, suggesting directions for developing more natural human-AI interactions. To this end, we utilize computational metrics that have been shown to capture humans’ real-time language processing. Specifically, surprisal ( Hale 2001 ; Levy 2008 ) , which reflects the unpredictability of a word given its context, has been linked to increased processing effort and has been shown to predict reading times ( Smith and Levy 2013 ) and N400 amplitudes ( Krieger et al. 2024 ; Salicchi and Hsu 2025 ) . Entropy, which captures the uncertainty within the probability distribution of upcoming stimuli, was suggested to be associated with P600 amplitudes ( Salicchi and Hsu 2025 ) . In this research, we investigate whether LALMs align with human cognitive mechanisms in social-linguistic processing. We use the EEG data from Wu and Cai 2024b as a benchmark of human processing and examine: a) whether LALMs align with humans in perceiving speaker characteristics and use them to guide real-time language processing; b) whether LALMs align with humans in the specific mechanism in processing speaker-content relationships. 2 Method 2.1 Human EEG data The human data were EEG responses to speech stimuli from native Mandarin Chinese speakers. The study employed a 2×2 factorial design crossing Congruency (speaker-congruent vs. speaker-incongruent) with Type (social vs. biological). Congruency was manipulated by matching or mismatching speaker characteristics with the sentence content, while Type distinguished between violations of social stereotypes and biological knowledge. The experimental materials consisted of 80 self-referential sentences (each with a speaker-congruent and a speaker-incongruent audio version) in Mandarin Chinese, with speaker characteristics varying along gender and age dimensions (Table 1 ). All sentence audios were generated using text-to-speech technique with consistent acoustic properties. Category Example English translation SM 在工作单位我一般都是穿 西服 打领带。 At the workplace I usually wear a suit and a tie. SF 这个周末我要先去做 美甲 然后理发。 This weekend I’m going to get a manicure and then a haircut. SA 我最近 上班 压力太大需要休息。 I’ve been working too hard lately and I need a break. SC 他把我的 玩具 抢走了我要去找妈妈告状。 He