A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook

Kaiwen Luo, Zhenhong Zhou, Leo Wang, Liang Lin, Yang Xiao, Tianyu Shao, Yuanhe Zhang, Yuxuan Li, Miao Yu, Kailin Lyu, Jiaming Zhang, Dongrui Liu, Li Sun, Yueming Wu, Kai Li, Ting Dang, Xiaojun Jia, Rohan Kumar Das, Xinfeng Li, Siyuan Liang, Qiufeng Wang, Xingjun Ma, Jing Chen, Kun Wang Junhao Dong, Deqing Zou, Yu Cheng, Xia Hu, Zhigang Zeng · 2026

Read A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook by Kaiwen Luo, Zhenhong Zhou, Leo Wang, Liang Lin, Yang Xiao, Tianyu Shao, Yuanhe Zhang, Yuxuan Li, Miao Yu, Kailin Lyu, Jiaming Zhang, Dongrui Liu, Li Sun, Yueming Wu, Kai Li, Ting Dang, Xiaojun Jia, Rohan Kumar Das, Xinfeng Li, Siyuan Liang, Qiufeng Wang, Xingjun Ma, Jing Chen, Kun Wang Junhao Dong, Deqing Zou, Yu Cheng, Xia Hu, Zhigang Zeng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20266.txt.

Introduction

Kaiwen Luo and thirty-four co-workers survey large audio language models. The survey studies how they work, detailing design ideas and alignment methods. It builds a trustworthiness taxonomy covering break-ins across modes, latent acoustic backdoors, and biometric privacy leakage. It reviews leading models through six pillars: hallucination, robustness, safety, privacy, fairness, and login checks. The imbalance between a mature offensive landscape and weak defenses highlights gaps and wide risks. It offers a roadmap with layered defenses, causal auditory world models, and intrinsic engineering.

Catalog record

CREATORKaiwen Luo, Zhenhong Zhou, Leo Wang, Liang Lin, Yang Xiao, Tianyu Shao, Yuanhe Zhang, Yuxuan Li, Miao Yu, Kailin Lyu, Jiaming Zhang, Dongrui Liu, Li Sun, Yueming Wu, Kai Li, Ting Dang, Xiaojun Jia, Rohan Kumar Das, Xinfeng Li, Siyuan Liang, Qiufeng Wang, Xingjun Ma, Jing Chen, Kun Wang Junhao Dong, Deqing Zou, Yu Cheng, Xia Hu, Zhigang Zeng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82377
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20266

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook Thanks: These authors contributed equally. Kaiwen Luo, Zhenhong Zhou, Leo Wang, Liang Lin, Yang Xiao, Tianyu Shao, Yuanhe Zhang, Yuxuan Li, Miao Yu, Kailin Lyu, Jiaming Zhang, Dongrui Liu, Li Sun, Yueming Wu, Kai Li, Ting Dang, Xiaojun Jia, Rohan Kumar Das, Xinfeng Li, Siyuan Liang, Qiufeng Wang, Xingjun Ma, Jing Chen, Kun Wang Junhao Dong, Deqing Zou, Yu Cheng, Xia Hu, Zhigang Zeng Abstract The foundational capabilities established by Large Language Models (LLMs) have paved the way for Multimodal Large Language Models (MLLMs), within which Large Audio Language Models (LALMs) are essential for realizing universal auditory intelligence. Despite their remarkable performance, the escalation of LALMs’ capabilities has significantly outpaced the development of systemic frameworks to ensure their trustworthiness. This survey provides a comprehensive investigation into the endogenous mechanisms of LALMs, detailing the architectural innovations and alignment algorithms that facilitate emergent reasoning. Specifically, we analyze how the transition to unified end-to-end frameworks and the integration of continuous acoustic signals inherently expand the attack surface. To rigorously evaluate the risks within these paradigms, we establish a comprehensive taxonomy of trustworthiness, categorizing critical vulnerabilities such as cross-modal jailbreaking, latent acoustic backdoors, and biometric privacy leakage. We review the state-of-the-art through six analytical pillars: hallucination, robustness, safety, privacy, fairness, and authentication. The profound imbalance between a mature offensive landscape and underdeveloped defenses further validates the critical trustworthiness gaps and multidimensional risks facing audio-centric intelligence. Finally, we propose a strategic roadmap advocating for ”Defense-in-Depth” architectures, causal auditory world modeling, and intrinsic representation engineering to bridge the gap between empirical performance and intrinsically trustworthy audio intelligence. Our project has been uploaded to GitHub https://github.com/Kwwwww74/Awesome-Trustworthy-AudioLLMs . See pages 1 of abstract I Introduction The emergence of Large Language Models (LLMs) [ 119 , 1 , 145 , 6 , 93 , 43 ] has transformed the landscape of artificial intelligence, establishing a robust foundation for the transition toward unified multimodal frameworks. This evolution into Multimodal Large Language Models (MLLMs) [ 179 , 178 , 142 ] is designed to emulate the multi-sensory nature of human perception across diverse sensory inputs. Among human senses, audio represents a primary medium for human communication and perception of the environment [ 75 ] , as it carries a vast amount of information within its signal. Previous research in audio intelligence relied on modular systems designed for a single task, such as automatic speech recognition [ 155 , 136 ] or sound classification [ 36 , 70 ] . Latest transition from these artifacts to unified Large Audio Language Models (LALMs) [ 23 , 141 , 128 , 22 , 166 ] represents a step for universal audio intelligence. Despite these remarkable advancements in auditory capabilities, the organic integration of language and audio modalities introduces complex safety and alignment challenges. Textual LLMs primarily address vulnerabilities within discrete text [ 134 , 154 , 190 , 107 ] . In contrast, LALMs introduce the audio modality, which presents a intricate risk landscape [ 89 , 13 , 3 , 16 ] due to the continuous properties of the acoustic signal. The deployment of LALMs within critical sectors further expands this complex risk landscape, translating these continuous-signal vulnerabilities into real-world threats. However, while the development of these capabilities is expanding, the research landscape remains fragmented and lacks a unified roadmap. Existing research predominantly details architectural innovations [ 130 , 2 , 189 ] or specific concerns [ 105 , 84 , 114 ] , yet there remains a significant lack of work dedicated to a systematic taxonomy of the safety implications for these systems. Recognizing that intrinsic trustworthiness cannot be guaranteed without a deep understanding of the underlying architecture, this research fragmentation highlights the necessity for a structured review that bridges the gap between mechanisms and safety. While foundational overviews and reviews of speech models [ 75 , 121 , 139 ] offer comprehensive insights into auditory perception, they often treat safety and ethical considerations as peripheral topics. Similarly, recent literature focused on evaluation provides a framework [ 182 ] for assessing model behavior but lacks a systematic taxonomy of the underlying security threats and safety mechanisms. Although an earlier review has addressed trustworthiness in speech [ 34 ] , they precede the recent shift toward unified generative frameworks, focusing largely on traditional machine learning. And specialized surveys remain predominantly concentrated on singular issues such as the detection of deepfakes and biometric authentication [ 188 , 81 , 124 ] . A comparison with these existing audio surveys is provided in Table I , illustrating the lack of literature dedicated to the implications of trustworthiness of these models. TABLE I: Comparison with existing surveys. Survey Obj. ‡ Trustworthiness † Stage ⋆ O H P F S R A D P F D E Year 2022 Feng et al. [ 34 ] S ✗ ✓ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✗ ✓ ✓ Year 2023 Latif et al. [ 75 ] A ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Yi et al. [ 188 ] S ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✓ ✓ Year 2024 Li et al. [ 81 ] S ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✓ ✗ Pham et al. [ 124 ] S ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ Peng et al. [ 121 ] A ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ Year 2025 Su et al. [ 139 ] A ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ Cui et al. [ 24 ] S+M ✓ ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✓ ✓ ✓ Yang et al. [ 182 ] A+M ✓ ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✓ Ours A+S+M ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ‡ Object : Audio-LLM (A), Speech-LM (S), Multi-modal LLM (M). † Trustworthiness : Hallucination (H), Privacy (P), Fairness (F), Safety (S), Robustness (R), Authentication (A). ⋆ Stage : Data Prep (D), Pre-training (P), Fine-tuning (F), Deployment (D), Evaluation (E). Outook : Outlook(O) Upon reviewing the aforementioned survey and systematically investigating the related literature, we conclude that our survey endeavors to address several questions that existing surveys have not covered. The main contributions of this survey are summarized as follows: • Systematic Investigation of Endogenous Mechanisms: We conduct a thorough examination of the internal structures within LALMs, detailing the structural improvements and alignment techniques that support the emergence of logical reasoning. This analysis provides the technical foundation required to understand the evolution toward unified models for auditory intelligence. • Comprehensive Trustworthy Review: We establish a systematic classification of trustworthiness challenges, identifying critical vulnerabilities including cross-modal jailbreak through acoustic cues, latent acoustic backdoors, and biometric privacy leakage. Additionally, we evaluate the landscape of current leading models through the six pillars of trustworthiness, which consist of hallucination, robustness, safety, privacy, fairness, and authentication. • Identification of Imbalance and Future Framework: Our analysis reveals a significant imbalance where offensive research has advanced significantly while defensive mechanisms remain limited and reactive.

Read the complete text