Comprehensive Audio Query Handling System with Integrated Expert Models and Contextual Understanding
Vakada Naveen, Arvind Krishna Sridhar, Yinyi Guo, Erik Visser · 2024
Read Comprehensive Audio Query Handling System with Integrated Expert Models and Contextual Understanding by Vakada Naveen, Arvind Krishna Sridhar, Yinyi Guo, Erik Visser (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03980.txt.
Introduction
Vakada Naveen, Arvind Krishna Sridhar, Yinyi Guo, and Erik Visser built a chatbot that answers questions about sound. Their twenty twenty-four paper links many expert models in one system. A query may ask what was said, who spoke, what song played, or for new audio to be made.
A classifier first reads the intent of the query and sends it to the right expert. A context module pulls events and times from the audio, and a language model of modest size blends all outputs into one reply. On custom tests of time-based questions, the system beat large audio models. It also did well on a known sound bench, which makes it a good fit for use on a device.
Catalog record
| CREATOR | Vakada Naveen, Arvind Krishna Sridhar, Yinyi Guo, Erik Visser |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03980
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Comprehensive Audio Query Handling System with Integrated Expert Models and Contextual Understanding Vakada Naveen, Arvind Krishna Sridhar, Yinyi Guo, Erik Visser Abstract This paper presents a comprehensive chatbot system designed to handle a wide range of audio-related queries by integrating multiple specialized audio processing models. The proposed system uses an intent classifier, trained on a diverse audio query dataset, to route queries about audio content to expert models such as Automatic Speech Recognition (ASR), Speaker Diarization, Music Identification, and Text-to-Audio generation. A 3.8 B LLM model then takes inputs from an Audio Context Detection (ACD) module extracting audio event information from the audio and post processes text domain outputs from the expert models to compute the final response to the user. We evaluated the system on custom audio tasks and MMAU sound set benchmarks. The custom datasets were motivated by target use cases not covered in industry benchmarks and included ACD-timestamp-QA (Question Answering) as well as ACD-temporal-QA datasets to evaluate timestamp and temporal reasoning questions, respectively. First we determined that a BERT based Intent Classifier outperforms LLM-fewshot intent classifier in routing queries. Experiments further show that our approach significantly improves accuracy on some custom tasks compared to state-of-the-art Large Audio Language Models and outperforms models in the 7B parameter size range on the sound testset of the MMAU benchmark, thereby offering an attractive option for on device deployment. 1 Introduction The rapid advancement of LLMs has significantly transformed the capabilities of chatbot systems, particularly in handling text-based queries Kumar et al. (2023) . However, the domain of audio content related queries remains relatively underexplored, with existing chatbots often limited to specific audio tasks Microsoft (2015) ; Google (2010) ; AWS (2017) ; Chu et al. (2023) ; Tang et al. (2023) . There is an increasing demand for intelligent systems capable of processing and understanding audio data in various contexts Zhao et al. (2019) . Whether it is recognizing music tracks, transcribing spoken language, or identifying speakers in a conversation, the ability to accurately interpret audio inputs is crucial for enhancing user interaction and satisfaction. Traditional chatbots like LUIS Microsoft (2015) , DialogFlow Google (2010) and Lex AWS (2017) , which primarily focus on speech or text, fail to meet these needs, highlighting the necessity for a more comprehensive approach. Even the latest multimodal LLMs are targeted towards specific speech and general audio related tasks Tang et al. (2023) ; Chu et al. (2023) ; Gong et al. (2023) ; Wu et al. (2023) fail to answer diverse audio queries. This paper introduces a novel chatbot system designed to address a broad spectrum of audio-related queries by integrating multiple specialized audio processing models. Our goal is to create a versatile and robust solution that surpasses the limitations of current systems. To this end, we first developed an intent classifier that effectively routes user queries to the appropriate audio expert models. This classifier was trained on a diverse dataset of audio-related questions, ensuring it can handle a wide range of queries with high accuracy. By leveraging advanced models such as Automatic Speech Recognition (ASR) Malik et al. (2021) , Speaker Diarization Park et al. (2022) , Music Identification Chaouch et al. (2020) , and Text to Audio generation Huang et al. (2023) , our system can process and respond to complex audio queries that require a combination of one or more of these expert models. Moreover, our system incorporates language models to provide coherent and contextually relevant responses. These models integrate text outputs from the audio expert models with additional context from the Audio Context Detection (ACD) model Kong et al. (2020) , which predicts audio events present in an audio along with their timestamps. This integration is particularly beneficial for handling multifaceted queries that require a deep understanding of the audio context. Experimental results highlight the effectiveness of our approach. A BERT-based intent classifier is shown to outperform a LLM-Fewshot intent classifier in terms of precision, recall, and F1-score. Furthermore, incorporating explicit ACD metadata in JSON format as inputs to the LLM significantly boosts the system’s accuracy. To gauge the system’s ability to address target use cases outlined in the paper, we also propose 2 new temporal task datasets and assess temporal reasoning skills using various methods, including zeroshot and few-shot prompting with both ground truth and predicted ACD audio events Finally, we compare our model with other large audio language models on the sound test of the MMAU benchmark Sakshi et al. (2024) , demonstrating its competitive performance despite lower complexity. To our knowledge, we are the first to propose a comprehensive solution for audio-related queries by integrating specialized audio processing and advanced language models. In addition we introduce a novel audio intent dataset for training a BERT based query intent classifier and assess system performance on two benchmark datasets for temporal reasoning tasks beyond what is covered in MMAU sound set. 2 Related Works / Background 2.1 Large Audio Language Models Recent studies have explored the integration of speech signals into large language models (LLMs), enabling them to directly process and understand general audio inputs. SALMONN Tang et al. (2023) is a speech audio language music open neural network that integrates a pre-trained text-based LLM with speech and audio encoders into a single multimodal model, achieving competitive performance on various speech and audio tasks. Similarly, the Qwen-Audio Chu et al. (2023) model scales up pre-training to cover over 30 tasks and various audio types, enhancing interaction capabilities with Qwen-Audio-Chat for multi-turn dialogues. The LTU (Listen, Think, and Understand) model Gong et al. (2023) introduces a new approach by combining audio perception with reasoning abilities, trained on the OpenAQA-5M dataset to exhibit strong performance on classification and captioning tasks, as well as emerging audio reasoning and comprehension abilities. GAMA, a general purpose Large Audio-Language Model (LALM) Ghosh et al. (2024) , integrates multiple audio representations and fine-tunes on a large-scale audio-language dataset, augmented with complex reasoning abilities through instruction-tuning with a synthetically generated CompA-R (Instruction-Tuning for Complex Audio Reasoning) dataset, demonstrating advanced audio understanding and reasoning capabilities. Another approach to integrating speech signals into LLMs is the use of decoder-only architectures, such as Speech-LLaMA Wu et al. (2023) . This model leverages a decoder-only architecture to map compressed acoustic features to the continuous semantic space of the LLM, demonstrating a significant improvement over strong baselines on multilingual speech-to-text translation tasks. Furthermore, LauraGPT Du et al. (2023) is a unified audio-and-text GPT-based LLM that can process both audio and text inputs and generate outputs in either modalities.These advances in audio-text integration demonstrate the potential of LLMs in handling audio-related tasks. 2.2 Chatbot Systems Our work builds upon the recent advances in large language models (LLMs) and their applications in various domains, particularly in the area of multimodal reasoning and action. Recent studies have shown that multimodal LLMs can be used for various audio and vision tasks, such as audio generation and editing Liang et al. (2024) , speech recognition Huang et al. (2024) , and image generation Surís et al. (2023) . Additionally, the concept of generalist agents, which aim to combine basic skills to solve complex tasks Ge et al. (2024) , has been explored in the context of