Causal Tracing of Audio-Text Fusion in Large Audio Language Models

Wei-Chih, Chien-yu, Hung-yi · 2026

Read Causal Tracing of Audio-Text Fusion in Large Audio Language Models by Wei-Chih, Chien-yu, Hung-yi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13768.txt.

Introduction

Wei-Chih Chen, Chien-yu Huang, and Hung-yi Lee use causal tracing to study audio-text fusion in large audio language models. How and where these models integrate acoustic features with textual context was unclear. They adapt causal tracing with layer-wise and token-wise analyses across DeSTA, Qwen, and Voxtral models. Layer-wise results show varied fusion strategies, from progressive integration to abrupt late-stage fusion. Token-wise results show the final sequence token acts as an informational bottleneck for audio retrieval.

Catalog record

CREATORWei-Chih, Chien-yu, Hung-yi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21651
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13768

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Causal Tracing of Audio-Text Fusion in Large Audio Language Models Wei-Chih, Chien-yu, Hung-yi Abstract Despite the strong performance of large audio language models (LALMs) in various tasks, exactly how and where they integrate acoustic features with textual context remains unclear. We adapt causal tracing to investigate the internal information flow of LALMs during audio comprehension. By conducting layer-wise and token-wise analyses across DeSTA, Qwen, and Voxtral, we evaluate the causal effects of individual hidden states. Layer-wise analysis identifies different fusion strategies, from progressive integration in DeSTA to abrupt late-stage fusion in Qwen. Token-wise analysis shows that the final sequence token acts as an informational bottleneck where the network decisively retrieves relevant information from the audio. We also observe an attention-like query mechanism at intermediate token positions that triggers the model to pull task-relevant audio context. These findings provide a clear characterization of when and where multi-modal integration occurs within LALMs. Chen Huang Lee keywords Large Audio Language Models, Internal Mechanism, Interpretability, Causal Tracing † † address: 1 National Taiwan University, Taiwan 2 Carnegie Mellon University, United States † † email: [email protected] 1 Introduction Recent progress in large language models (LLMs) [ 16 , 4 , 46 , 47 , 45 , 28 ] has enabled multi-modal systems to perceive and reason over various sensory inputs. Among these, Large Audio Language Models (LALMs) [ 29 , 30 , 12 , 11 , 27 , 15 , 43 , 19 ] now achieve strong performance on tasks ranging from speech recognition and captioning [ 17 , 49 , 20 ] to complex audio-aware question answering [ 31 , 40 ] . These models typically integrate audio encoders [ 37 , 7 ] with LLM backbones [ 16 , 4 , 46 , 47 , 45 , 28 ] to jointly process continuous audio streams and text inputs, mapping auditory features directly into the linguistic embedding space of the LLM. Despite this empirical success, the internal mechanisms driving this multi-modal integration remain unclear. Current research primarily focuses on architectural design and macroscopic task-level evaluations, such as overall performance [ 17 , 21 , 20 , 50 , 40 , 49 , 39 , 31 ] , bias [ 26 , 25 ] , and hallucination [ 22 , 8 ] . As a result, the exact dynamics of audio-text fusion are still a black-box. We currently lack a precise understanding of when across the model’s depth this integration occurs, such as early versus late-stage fusion, and where across the input sequence the network causally bridges auditory inputs with textual reasoning. Uncovering these dynamics is essential for improving model interpretability [ 32 ] , mitigating hallucinations [ 1 ] , and guiding architectural design [ 33 ] . To investigate how these models function internally, mechanistic interpretability provides interventional techniques such as causal tracing to map hidden information flow [ 32 ] . While these methods have successfully traced reasoning pathways in text models [ 32 , 51 ] and visual alignments in vision-language systems [ 34 , 23 ] , their application to large audio language models remains limited. Extending these tools to track how auditory attributes are integrated with textual context is a critical next step for understanding multi-modal reasoning. This paper investigates these internal dynamics by systematically adapting causal tracing to track how acoustic features are incorporated into linguistic representations. We propose a framework utilizing both spatial and depthwise analyses to evaluate the causal effects of specific hidden states during audio comprehension. By tracing the flow of information across different layers and token positions, we characterize the distinct pathways the network employs to bridge input audio with its textual output. We conduct experiments across several model families, including DeSTA, Qwen, and Voxtral. The results show that: (1) we identify divergent fusion strategies, ranging from progressive integration in DeSTA architectures to polarized late-stage fusion in Qwen models (Sec. 5.1 ); (2) the final sequence token acts as a critical informational bottleneck where the network retrieves relevant information from the audio to generate its response (Sec. 5.2 ); (3) an attention-like query mechanism exists at intermediate token positions that triggers the model to extract specific acoustic features (Sec. 5.2 ); and (4) the architectural divergences observed offer actionable insights for future model design and deployment (Sec. 5.3 ). Figure 1 : Illustration of our tracing methodology. The clean run (left) processes the original audio, while the corrupted run (right) replaces the audio with silence. By patching specific hidden states from the clean run into the corrupted run (e.g., layer-wise or token-wise), we isolate the causal contribution of individual layers and sequence positions to the final prediction. 2 Related Works 2.1 Probing Speech and Audio Models Several studies [ 38 , 5 , 35 , 36 , 10 , 24 , 44 ] have investigated the internal mechanics of speech and audio models, ranging from early architectures [ 42 , 2 ] to self-supervised learning frameworks [ 18 , 6 , 41 , 3 , 13 ] . Detailed layer-wise [ 35 , 36 , 10 ] and neuron-wise analyses [ 24 , 44 ] have been widely used to understand how these models process acoustic features. While the internal dynamics of these unimodal models are well documented, modern investigations into LALMs [ 17 , 21 , 20 , 50 , 40 , 49 , 39 , 31 ] rarely examine their internal working mechanisms. Instead of analyzing the cross-modal fusion process, current research predominantly focuses on task-level performances. Recent benchmarks primarily assess overall performance [ 17 , 21 , 20 , 50 , 40 , 49 , 39 , 31 ] , model bias [ 26 , 25 ] , or hallucination [ 22 ] based solely on the final generated text. Since these evaluations treat the LALM as an end-to-end black-box, the fundamental question of how and where acoustic features integrate with text representations remains unanswered. 2.2 Mechanistic Interpretability and Causal Tracing Mechanistic interpretability investigates how neural networks produce predictions by analyzing their internal hidden states. These analyses generally fall into two methodological categories: observational and interventional. Observational methods evaluate hidden states without altering the model computation, revealing only correlations between representations and specific tasks. In contrast, interventional methods, such as causal tracing [ 51 ] , actively modify hidden states during the forward pass to establish direct causal relationships. This causal approach has been highly successful in natural language processing. For example, the ROME framework [ 32 ] utilizes hidden state interventions to isolate the exact feed-forward layers causally responsible for factual recall. These interventional tools have recently been extended to multimodal systems [ 34 , 23 , 14 , 9 ] . For vision language models, causal techniques have identified specific layers where visual representations integrate with textual context [ 9 ] . However, interpretability in the speech and audio domain still heavily relies on observational methods. For instance, AudioLens [ 48 ] employs the logitlens technique, which projects intermediate hidden states directly to the vocabulary space, to observe layer-wise representation shifts. While these observational methods provide valuable correlational insights, they cannot isolate the exact hidden states that causally drive multimodal integration. This paper addresses the limitation by adapting causal

Read the complete text