M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models

2025

Read M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19358.txt.

Introduction

Yejin Kwon and three co-authors present M3-SLU, a benchmark for speaker-attributed reasoning in multimodal language models. Models read speech and text well, yet they still struggle to track who said what, and when, across a multi-speaker, multi-turn conversation, a skill real meeting and dialogue tools need.

Built from four open corpora, the benchmark holds more than twelve thousand checked examples with audio, transcripts, and metadata. Its two tasks ask models to answer questions tied to specific speakers and to match utterances to their speakers. Baselines from both pipeline systems and end-to-end models show where current systems stand.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19358

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models Abstract We present M3-SLU, a new multimodal large language model (MLLM) benchmark for evaluating multi-speaker, multi-turn spoken language understanding. While recent models show strong performance in speech and text comprehension, they still struggle with speaker-attributed reasoning, the ability to understand who said what and when in natural conversations. M3-SLU is built from four open corpora (CHiME-6, MELD, MultiDialog, and AMI) and comprises over 12,000 validated instances with paired audio, transcripts, and metadata. It includes two tasks: (1) Speaker-Attributed Question Answering and (2) Speaker Attribution via Utterance Matching. We provide baseline results for both cascaded pipelines and end-to-end MLLMs, evaluated using an LLM-as-Judge and accuracy metrics. Results show that while models can capture what was said, they often fail to identify who said it, revealing a key gap in speaker-aware dialogue understanding. M3-SLU offers as a challenging benchmark to advance research in speaker-aware multimodal understanding. Keywords: Multi-Speaker Spoken Language Understanding, Speech-LLM Benchmark, Evaluation M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models Yejin Kwon, Taewoo Kang, Hyunsoo Yoon † , and Changouk Kim † Department of Industrial Engineering, Yonsei University Seoul, Republic of Korea {beckykwon, hs.yoon, kimco}@yonsei.ac.kr, [email protected] Abstract content 1. Introduction Multimodal Large Language Models (MLLMs) have begun to blur the boundary between modalities, that is, seeing, hearing, and reasoning. Advances in models such as Qwen3-Omni ( Xu et al. 2025 ) , Gemini 2.5 ( Comanici et al. 2025 ) , and GPT5 ( Wang et al. 2025b ) have demonstrated how far AI systems can extend their understanding beyond text, seamlessly integrating visual, auditory, and linguistic cues to perceive the world in richer ways. Particularly, Audio-Language Models (ALMs) that integrate auditory representations into large language models, including Qwen2-Audio ( Chu et al. 2024 ) , Audio Flamingo ( Goel et al. 2025 ) , and Voxtral ( Liu et al. 2025 ) , have recently achieved remarkable progress in bridging speech and language understanding. These models enable machines not only to transcribe speech but also to understand intent, summarize dialogues, and generate coherent responses. Despite their impressive multimodal capabilities, most existing MLLMs still assume single-speaker conditions, achieving strong performance in Automatic Speech Recognition (ASR) but leaving Speaker Diarization (SD) largely unaddressed ( Yin et al. 2025 ) . Yet, real-world conversations are far more complex than such single-speaker settings. Understanding “who spoke when and what” offers a more comprehensive and meaningful perspective on real-world conversations ( Gao et al. 2025 ) . In particular, in natural interactions, utterance sequences may be immediately continuous or may overlap temporally; overlapping talk often occurs around turn transitions and is characterized by backchannels, interruptions, and simultaneous first-starts ( Levinson and Torreira 2015 ; Knudsen et al. 2020 ; Schegloff 2000 ) . Figure 1: Overview of the M3-SLU Benchmark. Benchmarks SLURP VoiceBench MMSU MMAU AudioBench MSU-Bench M3-SLU (Ours) Speaker-Oriented X X X X X O O Multi-speaker X X X X X O O Speaker-attributed Reasoning X X X X X \triangle O Audio Source TTS+RPC TTS+RPC RPC RPC RPC RPC RPC Conversation Type Monologue Monologue Dialogue Dialogue Dialogue Dialogue Dialogue Conversation Length Short Short Short Short Short Short Long (Over 1 Min) Table 1: Comparison between M3-SLU and existing speech-language understanding benchmarks. Understanding "Who spoke When and What" in multi-party conversations is a crucial step toward socially intelligent AI. However, most existing speech benchmarks ( Chen et al. 2024 ; Yang et al. 2024 ; Sakshi et al. 2024 ) address speaker-related and general dialogue tasks together, without separately examining the distinct challenges of speaker-centric understanding in real conversations ( Wang et al. 2025c ) . To close the gap between current MLLM evaluation and real-world conversational complexity, we introduce the M3-SLU Benchmark ( M ulti-Speaker, M ulti-Turn, and M ulti-Modal S poken L anguage U nderstanding), as illustrated in Figure 1 . Our key contributions: • We constructed M3-SLU benchmark using four open multi-speaker corpora — CHiME-6 ( Watanabe et al. 2020 ) , MELD ( Poria et al. 2019 ) , MultiDialog ( Park et al. 2024 ) , and AMI ( Kraaij et al. 2005 ) , reflecting diverse acoustic conditions and conversational patterns such as overlaps and rapid turns. • We propose the M3-SLU benchmark and evaluation framework for MLLMs , designed to measure performance on two simple yet challenging tasks that can only be solved by correctly identifying the speaker . 2. Related Work 2.1. Speech Understanding Models Speech understanding has advanced beyond mere transcription toward comprehension of spoken language – capturing both meaning and paralinguistic cues such as prosody and tone. Earlier pipeline systems that linked ASR to NLP models often lost acoustic detail and propagated recognition errors, prompting a shift toward end-to-end architectures that map raw audio directly to semantic representations. This architectural shift has enhanced robustness and enabled deeper speech understanding, as shown by recent models such as SpeechGPT ( Zhang et al. 2023 ) , Salmonn ( Tang et al. 2023 ; Yu et al. 2025 ) , Glm-4-voice ( Zeng et al. 2024 ) , and Gemini ( Team et al. 2023 ) . These models typically follow two main approaches: (1) Using an audio adaptor, as in Audio Flamingo 3 ( Goel et al. 2025 ) and Voxtral ( Liu et al. 2025 ) , or (2) Directly combining an audio encoder with an LLM, as in Qwen2-Audio ( Chu et al. 2024 ) . This paradigm has also paved the way for the emergence of more specialized capabilities, such as the Speaker LM ( Yin et al. 2025 ) , which captures individual vocal signatures and uses them in reasoning, and the MT-LLM ( Meng et al. 2025 ) , designed to disentangle and process dialogue from concurrent speakers. Looking forward, the frontier of research is expanding into multimodal domains where models like GPT-5 ( Wang et al. 2025b ) and Qwen2.5-Omni ( Xu et al. 2025 ) fuse auditory streams with visual data to achieve a contextually richer, more human-aligned interaction. 2.2. Speech Understanding Benchmarks As the performance of Large Audio-Language Models (LALMs) has advanced, developing benchmarks that evaluate their complex speech understanding capabilities has become a major research focus. Early benchmarks in Table 1 such as SLURP ( Bastianelli et al. 2020 ) and VoiceBench ( Chen et al. 2024 ) primarily focused on single-turn, single-speaker intent classification using synthetic or short speech segments (TTS 1 1 1 TTS (Text-to-Speech): Speech audio is synthetically generated from written text using a text-to-speech engine. + RPC 2 2 2 RPC (Real/Recorded Speech Corpus): Speech audio is naturally recorded from human speakers in real environments. ), laying the groundwork for fundamental speech-language understanding. Subsequent datasets including MMAU ( Sakshi et al. 2024 ) , MMSU ( Wang et al. 2025a ) , and AudioBench ( Wang et al. 2024 ) expanded their evaluation scope to multi-turn audio-based question and tasks such as intent classification and emotion interpretation, aiming to assess more nuanced aspects of speech understanding. However these benchmarks, while often built from long recordings, evaluated only short conversational segments (typically under 30 seconds) and were mainly oriented toward intent, emotion, or speaker recognition rather than context-grounded reasoning. Moreover, they did

Read the complete text