SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information

Chih-Kai, Neo, Yen-Ting, Hung-yi · 2025

Read SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information by Chih-Kai, Neo, Yen-Ting, Hung-yi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13237.txt.

Introduction

Chih-Kai Yang and three co-authors benchmark audio-language models in a 2025 paper. These models extend text models to speech and sound, and their basic skills are well tested. Multi-hop reasoning, recalling and combining several facts heard in audio, was not. Their benchmark targets exactly that skill, using questions built on speech and audio information.

The results expose a clear weakness. Models often extract the needed facts correctly yet fail to combine them into the right answer. That gap points to a basic limit in how these models reason across sound and speech. The authors offer the benchmark as a resource to drive work on multimodal reasoning.

Catalog record

CREATORChih-Kai, Neo, Yen-Ting, Hung-yi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13237

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information Chih-Kai, Neo, Yen-Ting, Hung-yi Abstract Large audio-language models (LALMs) extend the large language models with multimodal understanding in speech, audio, etc. While their performances on speech and audio-processing tasks are extensively studied, their reasoning abilities remain underexplored. Particularly, their multi-hop reasoning, the ability to recall and integrate multiple facts, lacks systematic evaluation. Existing benchmarks focus on general speech and audio-processing tasks, conversational abilities, and fairness, but overlook this aspect. To bridge this gap, we introduce SAKURA, a benchmark assessing LALMs’ multi-hop reasoning based on speech and audio information. Results show that LALMs struggle to integrate speech/audio representations for multi-hop reasoning, even when they extract the relevant information correctly, highlighting a fundamental challenge in multimodal reasoning. Our findings expose a critical limitation in LALMs, offering insights and resources for future research. Yang Ho Piao Lee nocounter]National Taiwan UniversityTaiwan keywords Large audio-language model, multi-hop reasoning, benchmark † † email: [email protected], [email protected] 1 Introduction Figure 1: Overview of dataset construction and experiments. Question-answer pairs were generated by GPT-4o and manually verified. LALMs were evaluated on speech/audio-based vs. text-based reasoning, where text-based replaces auditory inputs with captions, assessing their ability to integrate auditory information. Icon source: https://www.flaticon.com Large language models (LLMs) [ 1 , 2 ] have revolutionized AI research, extending beyond natural language processing to domain-specific applications like computer vision [ 3 ] and speech processing [ 4 , 5 ] . This shift drives the rise of large multimodal models, e.g., large vision-language models (LVLMs) [ 6 , 7 ] and large audio-language models (LALMs) [ 8 , 9 , 10 , 11 , 12 , 13 , 7 ] , equipping LLMs with cross-modal understanding. Research has also explored LLMs’ fundamental reasoning abilities [ 14 , 15 , 16 ] , with multi-hop reasoning , the skill of recalling and connecting multiple facts, recognized as a key to complex reasoning [ 17 , 18 , 19 , 20 ] . Particularly, latent multi-hop reasoning [ 19 , 20 ] , which involves multi-hop reasoning over internal parameterized knowledge stored in LLMs, is crucial for efficiently utilizing learned knowledge without heavily relying on external retrieval. Multi-hop reasoning datasets also provide insights into LLMs’ internal mechanism [ 21 ] , underscoring the importance of studying this ability. Despite growing interest in reasoning, multi-hop reasoning of large audio-language models (LALMs) based on speech and audio remains unexplored. Existing benchmarks assess general speech and audio-processing performance [ 22 , 23 ] , conversational abilities [ 24 , 25 ] , and fairness/bias [ 26 , 27 ] , while studies on LALMs’ fundamental abilities [ 28 , 29 , 30 ] remain limited and do not specifically address multi-hop reasoning. To address this gap, we introduce SAKURA ( S peech and A udio-based Question-answering Benchmar k for M u lti-hop R e a soning of Large Audio-Language Models), a novel benchmark for systematic evaluation of LALMs’ multi-hop reasoning using speech and audio information. SAKURA has four tracks based on speech/audio attributes including speaker gender, spoken language, speaker emotion, and animal sound. Each track includes single-hop and multi-hop sub-tracks, both comprising 500 multiple-choice questions. Single-hop questions require direct perception of an attribute (e.g., “What is the animal in the sound ?” ), whereas multi-hop questions demand further reasoning based on the extracted information (e.g., “Which of the listed features aligns best with the physical appearance of the animal in the sound ?” ). This design compels models to recall and integrate the extracted auditory cues when tackling multi-hop questions. Figure 1 provides examples of single-hop and multi-hop reasoning paths. In total, SAKURA offers 4000 human-verified multiple-choice questions, providing not only a structured framework for evaluating LALMs’ speech/audio-based multi-hop reasoning but also a valuable resource for future research on their reasoning mechanisms. Through evaluating various LALMs [ 8 , 9 , 10 , 11 , 12 , 13 , 2 , 7 ] on SAKURA, we find that LALMs struggle to recognize certain attributes and fail in multi-hop reasoning even when recognizing accurately. Moreover, while LALMs excel in reasoning based on text, they fail to integrate latent speech/audio representations into the reasoning process , contradicting expectations for end-to-end LALMs that should internally unify speech/audio understanding with LLM reasoning capabilities. Overall, our contributions are: (1) introducing SAKURA, the first benchmark for systematically evaluating LALMs’ multi-hop reasoning, and (2) uncovering LALMs’ difficulties in integrating speech/audio for multi-hop reasoning. Relevant resources such as dataset, evaluation code, etc., can be found at https://github.com/ckyang1124/SAKURA . 2 Dataset construction 2.1 Overview SAKURA evaluates LALMs’ multi-hop reasoning with speech/audio information across four tracks, covering fundamental attributes: speaker gender (Gender), spoken language (Language), speaker emotion (Emotion), and animal sounds (Animal), chosen for their significance in speech/audio processing. Each track includes single-hop and multi-hop sub-tracks. Here, we define the single-hop reasoning of LALMs as the direct perception and extraction of speech/audio attributes, forming the foundation for speech/audio-based multi-hop reasoning. In single-hop sub-tracks, models must extract an attribute directly from a speech/audio input (e.g., identifying the animal sound). In multi-hop sub-tracks, models must use the extracted information for further reasoning (e.g., determining a physical trait that matches the identified animal). Each sub-track contains 500 multiple-choice questions, each consisting of (1) a speech/audio input, (2) a textual question with a fixed number of choices, and (3) a golden answer. The multiple-choice format was adopted to facilitate standardized assessment and comparison. Additionally, speech/audio inputs are shared across sub-tracks within each track, enabling controlled evaluation of reasoning relative to perception accuracy. Figure 1 provides a data example, and Table 1 shows key dataset statistics, showcasing its diversity without strong biases toward any particular ranges. Table 1: Statistics of SAKURA. “Avg”, “Std”, “Min”, and “Max” represent the average, standard deviation, minimum, and maximum of the measured quantities. Avg Std Min Max Speech/audio duration (s) 4.79 2.23 0.22 20.78 Instruction length (words) 31.32 12.57 8 66 2.2 Speech/audio sources We describe the speech/audio data sources for each track: • Gender : we randomly selected 500 samples from the testing split of the English subset of Common Voice 17.0 [ 31 ] , which contains over 20k hours of validated human speech in 124 languages. We balance the genders of the speakers to mitigate potential biases. • Language : we drew 500 samples from the testing splits of eight languages 1 1 1 English/German/Spanish/French/Italian/Chinese/Japanese/Korean in Common Voice 17.0, while balancing both the language distribution and the speaker genders. • Emotion : we randomly chose 500 samples from CREMA-D [ 32 ] and MELD [ 33 , 34 ] , both widely used for speech emotion recognition. Samples were evenly distributed across five emotions 2 2 2 Happy / Disgust / Sad / Fear / Angry while ensuring gender balance among speakers. • Animal : we collected 500 audio samples from ESC-50 [ 35 ] and the dataset curated by Şaşmaz et al. [ 36 ] . ESC-50 contains 5-second environmental sounds categorized into 50 categories, from which we selected samples from nine animal categories 3 3 3 Dog / Cat / Pig / Cow / Frog / Hen / Rooster / Sheep / Crow . We included additional

Read the complete text