AudioRAG: A Challenging Benchmark for Audio Reasoning and Information Retrieval Jingru Lin, Chen Zhang, Tianrui Wang, Haizhou Li Abstract Due to recent advancements in Large Audio-Language Models (LALMs) that demonstrate remarkable performance across a range of sound-, speech- and music-related tasks, there is a growing interest in proposing benchmarks to assess these models. Existing benchmarks generally focus only on reasoning with internal knowledge, neglecting real-world scenarios that require external information grounding. To bridge this gap, we introduce AudioRAG, a novel benchmark designed to evaluate audio-based reasoning augmented by information retrieval in realistic web environments. This benchmark comprises both LLM-generated and manually curated question-answer pairs. Our evaluations reveal that even the state-of-the-art LALMs struggle to answer these questions. We therefore propose an agentic pipeline that integrates audio reasoning with retrieval-augmented generation, providing a stronger baseline for future research. † † volume: 312 † † year: 2026 † † workshop: AAAI 2026 Workshop on Audio-AAAI † † editors: Tatsuya Komatsu, Keisuke Imoto, Xiaoxue Gao, Nobutaka Ono, Nancy F. Chen keywords Audio Reasoning, Information Retrieval 1 Introduction Advancements in Large Audio-Language Models (LALMs) highlight their potential to unify speech, sound and music understanding within a single multimodal framework 4 ; 13 ; 8 . These advanced LALMs exhibit impressive performance across a wide range of understanding and generation tasks involving speech, sound, and music, such as speech recognition 18 , music genre classification 19 , audio captioning 3 , etc. While these tasks assess foundational audio understanding, they largely emphasize perceptual recognition rather than complex reasoning, which characterizes more sophisticated forms of intelligence. Recently, there has been a growing research interest in exploring the capacity of LALMs to perform multi-hop or deep reasoning, and several benchmarks have been proposed for this purpose 28 ; 17 ; 21 . For example, SAKURA 28 evaluates multi-hop reasoning capability. MMAR 17 evaluates the LALMs’ deep reasoning capabilities, including expert-level perceptual understanding and multi-step inference. However, these benchmarks primarily assess reasoning over internal, parameterized knowledge only. In real-world scenarios, users often ask questions that extend beyond a model’s internal knowledge. For example, a user might inquire about the final score of a soccer match mentioned in a recent news broadcast. If such information is absent from the model’s training data, the model may generate an unfaithful response. This problem is often recognized as hallucination in large language model (LLM) research 11 . To alleviate hallucination , retrieval-augmented generation (RAG) has been widely adopted, as it grounds the model outputs in retrieved and verifiable knowledge 12 ; 7 . By coupling generation with retrieval, RAG effectively enhances factual accuracy and reduces reliance on internal memorization. Nevertheless, existing audio reasoning benchmarks have not yet accounted for scenarios where audio-based reasoning is combined with retrieval. To address this gap, we introduce AudioRAG, a new benchmark featuring challenging questions tailored for audio reasoning with information seeking under real-world web environments. AudioRAG is constructed based on both open-sourced datasets and human-collected data. For open-sourced datasets, we take one attribute of the audio, e.g., genre of the music, species of the animal, from the metadata and prompt LLMs to generate questions based on that attribute. The generated questions are designed to require both audio processing and information retrieval steps. To further diversify the dataset, we manually collect videos from online sources and extract their audio tracks. Based on the extracted audios, we curate multi-hop questions similar to the ones generated by LLMs from the open-sourced datasets. Based on the benchmark, we evaluate several state-of-the-art LALMs. Our results show that existing LALMs struggle to answer these audio-based multi-hop reasoning questions. This observation is consistent with findings from previous studies 17 ; 28 . Moreover, when lacking the necessary knowledge to answer the question, the LALMs provide hallucinated responses. To better address such challenges, we propose an agentic pipeline capable of performing both audio reasoning and information retrieval, serving as a strong baseline for future research. Overall, our contributions are: (1) Introducing AudioRAG, the first benchmark for systematically evaluating models’ multi-hop audio reasoning and information retrieval capabilities; (2) We evaluate LALMs on the benchmark, revealing that current LALMs struggle to answer the questions; (3) We develop an agentic pipeline that shows a relative improvement of up to 24.9%. Figure 1: Data construction process (left) and the agentic pipeline (right). 2 Methods Each question in our dataset consists of a multi-hop question based on an audio context and the corresponding answer. The question is either in text or audio format, while the context is always in audio format. The multi-hop question is either generated based on open-source datasets or manually curated by humans. The following sections give details about the constructed dataset. 2.1 Dataset Statistics We first examine many open-source datasets and select the following for our dataset construction: • Speech/Sound/Music : MMAU 21 , CinePile 20 , Multitask-National-Speech-Corpus 23 , FMA 6 , Jazznet 1 , MusicNet 22 , iNaturalist 2 • Text : CHEER 10 On top of these open-source datasets, we also collect additional audio from online sources, then manually curate multi-hop questions based on the collected audio. Overall, we collect and generate 500 samples and release them at https://github.com/jingru-lin/AudioRAG 1 1 1 Details about the original sample sources are also provided. . Table 1: Prompts for the LLM to generate the multi-hop questions. Here, [ATTRIBUTE] refers to an attribute of the audio, such as the genre of music, the source of a sound etc. A prompt example to generate the multi-hop questions You are good at creating multi-hop questions. Your task is to generate multi-hop questions based on the given information about [ATTRIBUTE]. **Requirements**: 1. Do not mention the given [ATTRIBUTE] directly. You should assume an audio is given and the [ATTRIBUTE] must be derived from the audio. You can refer to it indirectly (e.g., using pronouns). 2. If possible, create a question that might involve an information-retrieval step to answer the question. 3. The final question must arrive at a clear correct answer. 4. Phrase the question naturally so it reads like a real user query. 5. Provide an answer to the questions. 6. No questions about the recognition of [ATTRIBUTE] should be generated. Examples: [FEW-SHOT_EXAMPLES] 2.2 Question Generation We utilize GPT-4o to generate the multi-hop question-answer pairs. Specifically, we take one attribute of the audio from metadata of the open-source datasets, for example, genre of music, species of animals heard in the audio, transcriptions of speech, and prompt GPT-4o to generate multi-hop questions that might require external knowledge to answer. Few-shot examples are provided in the prompts. Different templates and few-shot examples are needed for the audio samples from different datasets. Here, we only give an example of the prompt to illustrate the idea, as shown in Table 1 . The generated question should resemble a true user question after hearing the audio. Figure 1 shows an example. In addition, we manually collect some data from online sources. These questions contain timely information and are less likely to be exposed to data contamination. Therefore, current models lack sufficient parametric knowledge about the information required to answer the question, which makes retrieving external information highly important. An example question is shown in Figure 1 . 2.3 Data Filtering We employ two filters: a) Question Validity Filter; b) Answer Correctness Filter. The first ensures the quality of the generated questions. Specifically, we employ both LLM and annotators to verify whether the question has one unique answer. Questions lacking a unique answer are removed. The second filter focuses on verifying the answer correctness. To ensure the correctness of answers generated, we employ LLM-based agents with a search tool that allows for up-to-date information retrieval when answering the questions. We provide the ground-truth audio attributes (text) that were used to generate the questions for agents to answer the question. In such a way, it reduces the noise introduced when processing the audio, and the agents can focus exclusively on answer correctness. One example is shown in Appendix A . When the agents’ answers differ from the originally generated ones, human annotators review those cases and the corresponding questions are either revised or discarded. 3 Agentic Pipeline Our initial results show that the most open-source Large Audio Language Models (LALMs) struggle to answer these multi-hop questions. There are two main reasons. Firstly, the LALMs, while capable of handling audio data, show weak text capabilities, including instruction following, question interpretation, etc. Therefore, most LALMs struggle to break down the non-straightforward multi-hop questions into solvable subtasks. Secondly, the LALMs lack the knowledge, especially timely information, to answer the question. To tackle these problems, we propose an LLM-based agentic pipeline. Our pipeline makes use of a text-based LLM to handle user queries and drive tool usage. This pipeline is integrated with two tools: an audio processing tool and a search tool. While the former compensate for the text capabilities that the large language models lack, the latter complement the model with external knowledge. We will introduce the details of the pipeline in the following sections. 3.1 Pipeline Details Our pipeline is based on WebThinker 15 , but we enhance it with audio-processing capabilities. Given a user query, consisting of a question q that is either in text or audio form and an audio context c , the pipeline generates a solution for answering the user’s query regarding the audio context, guided by an instruction I . To answer the complex, multi-hop queries, the pipeline implements an autonomous Think-Call-Answer strategy, which iteratively thinks upon the current status, calls relevant tools, and generates intermediate/final responses. The final solution comprises a reasoning chain \mathcal{R} and a final output y . The reasoning LLM in the pipeline orchestrates the whole process and autonomously invokes tools from an available set \mathcal{T} during its reasoning process. The whole process can be formalized as: \displaystyle P(R,y\mid I,q,\mathcal{T}) \displaystyle=\underbrace{\prod_{t=1}^{T_{r}}P(R_{t}\mid R_{ [AUDIO-RELATED QUERY] < end_audio_analysis > . Upon receiving the query, the pipeline will initiate the audio processing tool, which will then output a response O_{a} based on the instruction. 4 Experiments 4.1 Raw Models We evaluate several open- and closed-sourced models, including Qwen2.5-Omni 25 , Audio Flamingo 3 9 , Audio-Reasoner 24 , Baichuan-Omni 16 , Qwen3-Omni 26 and Gemini-2.5-Flash 5 . 4.2 Agentic Pipeline We use Qwen3-8B 27 as the reasoning LLM to drive the whole pipeline. The search tool \mathcal{T}_{\text{exp}} used is the Google search engine API. The audio processing tool is Qwen2.5-Omni 25 / Qwen3-Omni 26 . Both reasoning LLM and audio processing tool are served with vLLM 14 , with four A100 (40GB) GPUs each. 4.3 Main Results Table 2: Main results. Since the agentic pipeline consists of audio and text LLMs, the size is broken down into size of (audio) + (text). The size of Gemini-2.5-Flash is unknown as it is a closed-source model. The rest are open-source models. Model Size Accuracy (%) Raw Model Qwen2.5-Omni 25 7B 32.2 Audio Flamingo 3 9 8.3B 28.8 Audio-Reasoner 24 8.4B 20.2 Baichuan-Omni 16 11B 24.4 Qwen3-Omni 26 30B 37.0 Gemini-2.5-Flash 5 - 45.0 Agentic Pipeline Qwen2.5-Omni + Qwen3-8B 27 7B + 8B 39.5 Qwen3-Omni + Qwen3-8B 27 30B + 8B 46.2 Table 2 shows the performance of both raw models and agentic pipelines on the benchmark. The reported metric is accuracy (%), which records to number of correct questions answered. Each question is provided with an answer and GPT-4o is used to evaluate the generated answer. The evaluation prompt is shown in Appendix B . The results in Table 2 are averaged over three runs. From Table 2 , the strongest raw model is the closed-source Gemini-2.5-Flash, achieving 45% accuracy, followed by open-sourced Qwen3-Omni with 37.0%. All other open-sourced models perform notably worse, highlighting a clear performance gap between open-source and closed-source models. This is likely due to the latter’s stronger internal knowledge or agentic abilities. Among all open-sourced models, the Qwen family consistently outperforms others, demonstrating stronger audio reasoning capabilities. Compared to the raw models, the agentic pipeline consistently delivers superior performance. Specifically, Qwen2.5-Omni and Qwen3-Omni achieve accuracies of 32.2% and 37.0%, respectively. When integrated into the agentic pipeline with Qwen3-8B as the reasoning model, their accuracies rise to 39.5% and 46.2%, corresponding to relative improvements of 22.7% and 24.9%. Table 3: Case study comparing outputs from a raw model and agentic pipeline. Reasoning error is highlighted in red. Audio: an audio clip about “The Wizard of Oz” Question: How is the movie mentioned in this audio related to Wicked? Answer by Qwen3-Omni : The movie mentioned is **The Wizard of Oz**. \textcolor redThe audio does not mention the movie **Wicked** at all. Answer by Agentic Pipeline with Qwen3-Omni : Wicked is a prequel to The Wizard of Oz, exploring… Ground-truth Answer : Wicked is a prequel to The Wizard of Oz Table 3 shows an example of actual outputs from a raw model and an agentic pipeline. The question in this example requires reasoning beyond the information directly available in the audio content. The raw model focuses purely on the audio input, correctly identifying the mentioned movie but failing to infer the link between the two movies. In contrast, the agentic pipeline demonstrates a clearer understanding of the question and effectively retrieves external knowledge to identify the correct relationship between the two movies. Table 2 and 3 both show that the agentic pipeline bridges the gap in reasoning and knowledge integration between raw models and human-like multimodal understanding. While the raw models demonstrate perceptual capabilities, their performance is limited by a lack of contextual reasoning and external knowledge grounding. The agentic pipeline addresses these limitations. Figure 2: Error types breakdown for incorrect answers. The x-axis is the category of errors. A refers to Reasoning Error, B refers to Audio Processing Error, C refers to Knowledge Error and D refers to Invalid Answer. 4.4 Analysis on Error Types In this section, we analyze the different error types made by the raw models and the agentic pipelines. We summarise four error types: a) Reasoning Error, where the model fails to understand the question or apply correct reasoning to arrive at the answer; b) Audio Processing Error, where the question requires to recognise a specific attribute of the audio but the model wrongly recognise it; c) Knowledge Error, where the question requires external knowledge beyond what is in the audio but the model gives the wrong answer due to lack of or incorrect knowledge; d) Invalid Answer: the model gives no response or incomplete output. GPT-4o is used to analyze the answers and categorize the incorrect answers into one of these categories. Figure 2 shows the number of questions belonging to each error category. The evaluation prompt is shown in Appendix B . This plot is based on outputs from raw models Qwen2.5-7B and Qwen3-30B and their corresponding agentic pipeline coupled with Qwen3-8B. From Figure 2 , we observe that the agentic pipeline reduces Reasoning, Audio Processing and Knowledge Error, but tends to produce more Invalid Answer. The largest reduction is in Reasoning Error. This indicates that raw models often struggle to interpret complex multi-hop questions but the structured multi-step reasoning of the agentic pipeline significantly improves comprehension. The second major improvement lies in Knowledge Error. This is due to the knowledge retrieval component in the agentic pipeline which enhances the model with up-to-date and factually accurate information. Improvements in Audio Processing Error are smaller, as the pipeline still relies on the raw model to process audio. Invalid Answer increases, likely because the more complex multi-step reasoning can lead the pipeline into infinite logical loops. 5 Conclusion In this paper, we introduce AudioRAG, the first benchmark designed to evaluate models’ multi-hop audio reasoning and information retrieval capabilities. Our experiments show that current LALMs perform poorly on this challenging benchmark, while an agentic pipeline that integrates audio reasoning with information retrieval establishes a stronger baseline for future research. 6