DEAF: A Benchmark for D iagnostic E valuation of A coustic F aithfulness in Audio Language Models
Jiaqi Xiong, Yunjia Qi, Qi Cao, Yu Zheng, Yutong Zhang, Ziteng Wang, Ruofan Liao, Weisheng Xu, Sichen Liu, [0.1em] University of Oxford XJTLU HNU CUHK(SZ) PKU HKUST(GZ) [0.1em] [email protected] · 2026
Read DEAF: A Benchmark for D iagnostic E valuation of A coustic F aithfulness in Audio Language Models by Jiaqi Xiong, Yunjia Qi, Qi Cao, Yu Zheng, Yutong Zhang, Ziteng Wang, Ruofan Liao, Weisheng Xu, Sichen Liu, [0.1em] University of Oxford XJTLU HNU CUHK(SZ) PKU HKUST(GZ) [0.1em] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18048.txt.
Introduction
Xiong and co-authors present DEAF, a bench that tests whether audio language models truly hear sound. It holds over 2,700 conflict items across prosody, background sounds, and speaker identity. Test levels raise text influence step by step, from content clashes to misleading prompts. New metrics measure how much models lean on text over sound. Seven audio models show a clear pattern: text drives most calls, despite real sound sensitivity.
Catalog record
| CREATOR | Jiaqi Xiong, Yunjia Qi, Qi Cao, Yu Zheng, Yutong Zhang, Ziteng Wang, Ruofan Liao, Weisheng Xu, Sichen Liu, [0.1em] University of Oxford XJTLU HNU CUHK(SZ) PKU HKUST(GZ) [0.1em] [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39655 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18048
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DEAF: A Benchmark for D iagnostic E valuation of A coustic F aithfulness in Audio Language Models Jiaqi Xiong, Yunjia Qi, Qi Cao, Yu Zheng, Yutong Zhang, Ziteng Wang, Ruofan Liao, Weisheng Xu, Sichen Liu, [0.1em] University of Oxford XJTLU HNU CUHK(SZ) PKU HKUST(GZ) [0.1em] [email protected] Abstract Recent Audio Multimodal Large Language Models (Audio MLLMs) demonstrate impressive performance on speech benchmarks, yet it remains unclear whether these models genuinely process acoustic signals or rely on text-based semantic inference. To systematically study this question, we introduce DEAF ( D iagnostic E valuation of A coustic F aithfulness), a benchmark of over 2,700 conflict stimuli spanning three acoustic dimensions: emotional prosody, background sounds, and speaker identity. Then, we design a controlled multi-level evaluation framework that progressively increases textual influence, ranging from semantic conflicts in the content to misleading prompts and their combination, allowing us to disentangle content-driven bias from prompt-induced sycophancy. We further introduce diagnostic metrics to quantify model reliance on textual cues over acoustic signals. Our evaluation of seven Audio MLLMs reveals a consistent pattern of text dominance: models are sensitive to acoustic variations, yet predictions are predominantly driven by textual inputs, revealing a gap between high performance on standard speech benchmarks and genuine acoustic understanding. Figure 1: Illustration of the three acoustic–semantic conflict types in DEAF . ESC : the text expresses happiness while the vocal tone conveys depression. BSC : the text implies a quiet setting while the audio contains noisy traffic. SIC : the text implies an elderly female speaker while the voice belongs to a young boy. In each case, the correct answer requires following the audio signal, not the text. 1 Introduction Acoustic signals and lexical semantics are usually aligned in natural speech. However, critical paralinguistic information often resides in their occasional divergence, where the speaker’s voice contradicts the literal meaning of the words. This state of modality conflict, characterized by the divergence between acoustic cues and lexical semantics, serves as a rigorous litmus test for genuine audio understanding. While human listeners prioritize prosodic nuances to decode a speaker’s true intent such as sarcasm or hesitation, current Audio Multimodal Large Language Models (Audio MLLMs) may achieve high benchmark scores by merely exploiting semantic redundancies rather than performing authentic acoustic processing. This raises a fundamental research question: do Audio MLLMs perform authentic acoustic inference, or simply defer to the most probable textual interpretations. In the visual modality, this question has been studied extensively. Frank et al. 2021 demonstrate that multimodal transformers frequently rely on text while ignoring visual input, and Wang et al. 2026 confirm systematic text dominance in vision–language models through controlled cross-modal conflict evaluation. Interpretability analyses further reveal that cross-modal attention often collapses onto the language modality Aflalo et al. 2022 , and the “right for the wrong reasons” phenomenon McCoy et al. 2019 shows that high accuracy can mask reliance on spurious shortcuts. These findings raise an important concern for multimodal learning, but they primarily focus on the vision–language paradigm. Whether similar modality biases arise in audio-based multimodal models remains less understood. Recent Audio MLLMs Tang et al. 2024 ; Chu et al. 2024 ; Team et al. 2023 ; Hurst et al. 2024 achieve impressive results on speech emotion recognition, speaker identification, and acoustic scene classification Wang et al. 2024 ; yu Huang et al. 2024 ; Yang et al. 2024 ; Chen et al. 2024 . However, in all these benchmarks, acoustic features and semantic content are naturally aligned —a sad speaker says sad things, and kitchen sounds accompany talk of cooking. This alignment means that a model performing internal ASR followed by text reasoning would score just as well as one that genuinely processes acoustic signals, making it impossible to tell which strategy a model actually uses. Several concurrent studies have begun probing this gap specifically for emotion. The LISTEN benchmark Chen et al. 2025 introduces controlled emotion–semantic conflict conditions to disentangle lexical and acoustic reliance. The EMIS dataset Corrêa et al. 2025 synthesizes emotionally incongruent speech via TTS, revealing that models often prioritize semantic content over vocal cues. Huang et al. 2026 propose the FAS/CASE framework to explicitly disentangle acoustic and semantic pathways under emotional conflict. While these efforts provide valuable initial evidence of text bias, they share two critical limitations: (1) Narrow acoustic scope : all existing work confines conflict evaluation to emotional prosody, leaving it unknown whether text dominance generalizes to other acoustic dimensions such as background sounds and speaker characteristics; and (2) Single-condition design : by testing under only one conflict setting, these studies cannot disentangle whether model errors stem from semantic content bias within the audio or from sycophantic compliance with textual prompts—two fundamentally different failure modes that demand distinct mitigation strategies. To address these gaps, we introduce DEAF (Diagnostic Evaluation of Acoustic Faithfulness)(Figure 1 ). DEAF advances beyond prior work in three respects: • Multi-dimensional conflict coverage. We construct over 2,700 stimuli spanning three acoustic dimensions, namely motion-Semantic Conflict (ESC), Background Sound-Semantic Conflict (BSC), and Speaker Identity-Semantic Conflict (SIC), providing the first unified diagnostic beyond emotion alone. • Progressive textual interference. A three-level framework systematically increases textual interference: Level 1 presents acoustic–semantic conflict alone; Level 2 adds a misleading prompt; Level 3 combines both. This enables fine-grained attribution of errors to semantic content bias (L1 vs. L3) versus prompt sycophancy (L2 vs. L3). Within each level, we further vary explicit versus implicit semantic cues to test whether lexical specificity modulates the degree of text dominance. • Diagnostic metrics. We propose the Acoustic Robustness Score (ARS), which jointly requires sensitivity to acoustic variation and prediction correctness, and the Environment Discrimination Index (EDI) for measuring fine-grained background-sound discrimination. 2 The DEAF Benchmark 2.1 Three Conflict Types DEAF targets three independent, non-textual information layers in audio, each designed to test a distinct aspect of acoustic understanding. For each conflict type, we construct Matched (acoustic features align with semantic content; control condition) and Mismatched (acoustic features contradict semantic content; experimental condition) pairs. If a model answers identically under both conditions, it is only “reading” text; if its answers change with acoustic conditions, it is genuinely “hearing” the audio. Emotion-Semantic Conflict (ESC). ESC tests whether models detect vocal emotion independent of semantic content. We build on the EMIS dataset Corrêa et al. 2025 , which pairs 104 English sentences across three semantic conditions— Explicit (containing emotion words), Implicit (contextually emotional) as shown in Figure 2 —with four emotional prosodies (angry, happy, neutral, sad). (a) Happy – Explicit (b) Happy – Implicit Figure 2: Word clouds of the happy class under Explicit and Implicit semantic conditions in ESC. Figure 3: Overview of the DEAF dataset construction pipeline. Background Sound-Semantic Conflict (BSC). BSC tests whether models identify background sounds independent of what the speaker says. We construct stimuli by mixing neutral-prosody synthesized speech with real-world background sounds from the DEMAND noise database Thiemann et al. 2013 . We define 18 sub-environments grouped into six categories (Domestic, Nature, Office, Public, Street, Transportation; three sub-environments each; see Appendix A.2.2 ). Text scripts comprise 84 sentences: 72 environment-specific (36 explicit + 36 implicit) and 12 neutral. We design two mismatch granularities to probe discrimination ability: • Within-Mismatch : text and background come from different sub-environments