Do Audio-Visual Large Language Models Really See and Hear?

Ramaneswaran Selvakumar *, Kaousheik Jayakumar *, S Sakshi, Ruohan Gao #, Dinesh Manocha #, # Equal advising · 2026

Read Do Audio-Visual Large Language Models Really See and Hear? by Ramaneswaran Selvakumar *, Kaousheik Jayakumar *, S Sakshi, Ruohan Gao #, Dinesh Manocha #, # Equal advising (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02605.txt.

Introduction

Selvakumar and five colleagues run the first mechanistic study of audio-visual language models, tracing how audio and visual features fuse layer by layer into text. Rich audio meaning appears in middle layers but rarely reaches the final output when audio conflicts with vision, because deeper fusion layers favor visual signals. The audio behavior closely matches the vision-language base model, pointing to limited extra training on audio supervision.

Catalog record

CREATORRamaneswaran Selvakumar *, Kaousheik Jayakumar *, S Sakshi, Ruohan Gao #, Dinesh Manocha #, # Equal advising
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49052
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.02605

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Audio-Visual Large Language Models Really See and Hear? Ramaneswaran Selvakumar *, Kaousheik Jayakumar *, S Sakshi, Ruohan Gao #, Dinesh Manocha #, # Equal advising Abstract Audio-Visual Large Language Models (AVLLMs) are emerging as unified interfaces to multimodal perception. We present the first mechanistic interpretability study of AVLLMs, analyzing how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs. We find that although AVLLMs encode rich audio semantics at intermediate layers, these capabilities largely fail to surface in the final text generation when audio conflicts with vision. Probing analyses show that useful latent audio information is present, but deeper fusion layers disproportionately privilege visual representations that tend to suppress audio cues. We further trace this imbalance to training: the AVLLM’s audio behavior strongly matches its vision-language base model, indicating limited additional alignment to audio supervision. Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision. 1 Introduction Audio-Visual Large Language Models (AVLLMs) 56 ; 11 ; 51 ; 63 ; 35 extend large language models (LLMs) to process real-world multimodal inputs such as audio and video. Audio conveys information that vision alone cannot: off-screen events, speech, music, and ambient cues are primarily auditory. By combining these signals with the reasoning capabilities of LLMs, AVLLMs achieve a more holistic understanding of complex scenes–enabling applications in multimedia analysis, education 2 , human–computer interaction 10 ; 66 , robotics 21 ; 60 , healthcare 38 , and biodiversity monitoring 40 . Despite rapid progress, current AVLLMs remain black-boxes in terms of how they process and utilize audio-visual information. While interpretability has been explored for text-only LLMs 15 ; 18 ; 62 , vision-language 26 ; 42 ; 6 , and audio-language models 59 ; 47 ; 36 , the mechanisms of audio-visual integration have not been studied. This opacity raises reliability concerns, particularly in safety-critical settings— e.g . , an autonomous vehicle should respond to an out-of-view ambulance siren even when it is not visible. To illustrate this, consider Fig 1 : a scene where a blue car and a woman walking a dog are visible, while an ambulance siren is heard off-screen. When prompted to “describe what you see and hear,” existing AVLLMs frequently hallucinate sounds from visible but silent objects and ignore the siren. We observe that for current AVLLMs (Fig 2 ) relative performance drops by up to 56% on such counter-factual samples with conflicting modalities, as compared to factual samples where both modalities are aligned, indicating that current AVLLMs rely heavily on vision and underutilize audio cues. Figure 1 : Illustration of visual bias. AVLLMs exhibit a critical modality bias, often prioritizing visual cues over vital audio cues. The diagram illustrates a counterfactual scene, visible objects (a blue car and a woman walking a dog) are silent and the only audible sound is an out-of-view ambulance siren. When prompted to describe the scene, the AVLLM hallucinates audio events (car engine, dog barking) and misses the actual siren sound. To address these issues, we systematically analyze why AVLLMs fail to utilize audio inputs effectively. Most AVLLMs adopt an adapter-based architecture 8 , where pretrained audio and vision encoders feed representations through learned adapters into the LLM token space-extending designs such as Llava 33 . We focus on the LLM backbone, the largest and most influential component, to examine how audio and visual representations evolve, interact, and influence text generation across layers. Unlike prior interpretability work on single-modality models 26 ; 42 ; 6 ; 59 , AVLLMs present unique challenges: audio and vision interact through complex cross-modal pathways, and their complementarity makes isolating each modality’s contribution far more difficult. Figure 2 : Audio Understanding Performance. Audio understanding severely degrades under audio-visual conflict. Using this testbed, we perform a multi-stage mechanistic interpretability analysis to understand where and why audio information fails to manifest in generation: (1) We analyze attention patterns across layers to identify where AVLLMs allocate focus between modalities and whether audio receives sufficient attention. (2) We then probe audio representations to determine if they encode meaningful information. (3) To establish causal relationships between modality representations and outputs, we design attention knockout experiments that selectively block audio or visual pathways, allowing us to trace how each modality influences generation. (4) Finally, to investigate the origin of any observed bias, we compare output token distributions with base vision-language models, testing whether visual dominance stems from inherited training artifacts or architectural constraints. Together, these analyses provide the first mechanistic evidence of how and where modality imbalance arises in AVLLMs. To summarize, our main contributions are: 1. We present the first systematic mechanistic analysis of Audio-Visual Large Language Models, dissecting how audio and visual representations are encoded, integrated, and manifested in text generation. 2. We show that AVLLMs’ audio understanding drastically deteriorates by upto 56% when audio and visual cues conflict–revealing a strong modality preference despite architectural capacity for multimodal reasoning. 3. By probing audio token representations in intermediate layers we demonstrate that they evolve into interpretable onomatopoeic tokens which describe sound events, indicating that AVLLMs encode strong latent audio semantics that remain untapped at generation time. 4. Using causal mediation via attention knockouts, we find that deeper layers prioritize visual features and actively suppress audio information. Blocking visual signals at these layers restores audio reasoning, providing direct causal evidence of cross-modal interference. 5. We show that the output token distributions of AVLLMs strongly mirror those of their base vision-language models, suggesting that the visual bias potentially originates from its alignment tuning and data rather than architectural limitations. 2 Related Work Audio Visual Large Language Models (AVLLMs). AVLLMs aim to integrate auditory and visual perception in large language models, thus enabling reasoning across audio, vision, and text. Most AVLLMs follow an adapter-based architecture, popularized by Large Vision Language Models (LVLMs) 8 ; 7 where frozen audio and visual encoders are connected to a pretrained language backbone through learned projection modules (adapters). Early systems such as Video-LLaMA 63 and PandaGPT 50 adopt simple MLP adapters to map modality embeddings into the LLM token embedding space. More recent AVLLMs such as Qwen2.5-Omni 56 , Qwen3-Omni 58 , and MiniCPM 23 have dedicated modules that temporally align audio and visual features before mapping to the LLM embedding space, improving synchronization and contextual grounding. While most approaches build on top of base LLMs, others, such as InternOmni 25 build on existing LVLMs like InternVL 12 . This extension strategy improves training efficiency and leverages strong vision-language foundations, but may introduce visual bias from the underlying LVLM. Mechanistic Interpretability of LLMs. Mechanistic interpretability seeks to understand the internal mechanisms within neural networks. For LLMs, this has revealed how models store factual knowledge 41 ; 37 ; 17 , localize specific capabilities 54 ; 45 , and process information across layers 16 . While well-established for text-only models, mechanistic interpretability remains nascent for multimodal LLMs. 42 show that object information is localized in token representations and gradually aligns with language representations, while 26 and 65 trace cross-modal image token flow, revealing a distinct two-step processing pattern. Tools For Mechanistic Interpretability. Understanding and explaining neural networks, particularly

Read the complete text