Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

Neta, Lenny, Ethan · 2026

Read Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering by Neta, Lenny, Ethan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06854.txt.

Introduction

Glazer and colleagues ask whether audio models listen, tackling text bias where word priors override key audio evidence. With mechanistic probing, they find a set of audio-special heads whose focus yields a listening signal. That signal rises when audio evidence shapes model output, marking use under prompting. From that finding they build a steering direction from audio versus silence and apply an inference-time activation tweak to the output code, boosting audio effect. On MMAU, the tweak lifts scores on two Qwen-based models with no weight changes.

Catalog record

CREATORNeta, Lenny, Ethan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22602
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06854

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering Neta, Lenny, Ethan Abstract Multimodal large language models can exhibit text dominance, over-relying on linguistic priors instead of grounding predictions in non-text inputs. One example is large audio-language models (LALMs) where decisive audio evidence can be under-utilized even when it contains important information. To address this issue we use mechanistic interpretability to identify a small set of audio-specialist attention heads whose audio attention yields a “listening” signal. We show that this signal increases when audio evidence affects the model’s output, providing an indicator of audio engagement under standard prompting. Leveraging this localization, we construct an audio–silence steering direction and apply an inference-time activation intervention to the final representation, amplifying the model’s audio effect. To demonstrate the utility of this intervention, we show on MMAU that this improves accuracy by up to +8.0 percentage points on two Qwen-based LALMs, without any parameter updates. Glazer Aharon Fetaya keywords speech recognition, interpretability, large audio language models † † address: 1 Bar-Ilan University, Ramat Gan, Israel 2 Columbia University, New York, NY, USA † † email: [email protected], [email protected] 1 Introduction Large audio-language models (LALMs) couple a pretrained audio encoder with a decoder-based large language model (LLM), enabling instruction-following understanding and reasoning over speech, environmental sounds, and music from natural-language prompts [ 1 , 2 , 3 , 4 ] . Architecturally, common fusion strategies include (i) projecting audio-encoder representations into the LLM embedding space and inserting them as a short sequence of audio-conditioned pseudo-tokens processed jointly with text via self-attention [ 5 , 2 ] , and (ii) conditioning the LLM on audio features through added (often gated) cross-attention adaptor layers, rather than interleaving audio tokens with the text stream [ 3 ] . However, processing these mixed token streams within a single backbone trained predominantly on text introduces a critical phenomenon: text dominance . Even when non-text modalities are highly informative, models frequently rely disproportionately on linguistic cues. Recent systematic evidence shows that this imbalance is pervasive across modalities, including audio, and stems from factors like fusion design choices and attention dilution caused by non-text token redundancy [ 6 , 7 ] . Related work frames this phenomenon as language-prior bias , demonstrating that multimodal outputs are often driven more by the underlying LLM’s priors than by the non-text inputs themselves [ 8 ] . Specifically within the audio-language domain, controlled mismatch studies demonstrate that capable LALMs can be dominated by textual prompts, effectively disregarding contradictory audio evidence [ 9 , 10 ] . Mechanistic interpretability has become a central framework for analyzing the internal computations of text-only LLMs, aiming to identify localized mechanisms in weights and activations that causally drive model behavior rather than relying on post-hoc rationales [ 11 , 12 , 13 ] . More recently, this toolkit has begun to extend to multimodal architectures, including LALMs, enabling component-level analyses of how non-text modalities are integrated and where modality-specific failure modes arise [ 14 , 15 , 16 , 17 ] . A key part of this paradigm is the use of causal interventions on internal activations (e.g., ablation or activation patching) to test mechanistic hypotheses. Building on this idea, steering refers to inference-time interventions that modify internal activations to influence how information is processed [ 18 ] . A recurring finding is that transformer components, particularly individual attention heads, exhibit stable and specialized computational roles, enabling causal intervention at the head level and motivating activation-based steering approaches that modulate model behavior during inference [ 12 , 19 , 20 ] . Motivated by the text-dominance failure mode in LALMs and recent progress in mechanistic interpretability for multimodal transformers, we use mechanistic tools to study audio under-utilization in depth. In particular, we ask whether head-level signals can indicate when the model is engaging with audio, and whether these signals can be used as a practical handle for inference-time steering. In this work, we make two main contributions. First, we identify a small set of audio-specialist attention heads whose attention to audio is predictive of correctness, yielding an instance-level “listening” signal. Second, we demonstrate that mechanistic analysis at the component level can provide an actionable handle in audio-language models: using the localization to guide a controlled inference-time activation intervention, we amplify the model’s audio effect and improve MMAU performance for two Qwen-based LALMs (Qwen2-Audio-7B [ 1 ] and R1-AQA [ 21 ] ), without any parameter updates. Figure 1: Specialist-Guided Steering (SGS). (a) We identify audio-specialist attention heads by computing each head’s audio-attention share and selecting the Top- K heads whose audio attention is most predictive of correctness on a calibration set. (b) We run audio and matched-duration silence forward passes and form a layer-localized steering direction by aggregating residual differences (\mathbf{h}^{\text{aud}}_{\ell}-\mathbf{h}^{\text{sil}}_{\ell}) over the specialist layer set \mathcal{L} (layers containing the discovered heads). We scale this direction by \beta and add it to the final representation to obtain \mathbf{h}^{*} for prediction. 2 Related Work Text Dominance in Multi Modal LLMs. Large audio-language models (LALMs) extend instruction-following LLMs with audio front-ends and multimodal fusion mechanisms, such as token injection, enabling joint reasoning over audio and text [ 1 , 3 ] . However, a reliability concern in these systems is text dominance (or language-prior bias), where linguistic cues override informative non-text evidence [ 6 , 22 ] . This phenomenon has been systematically documented across multimodal settings, where models default to strong language priors even when they conflict with perceptual evidence, leading to spurious correlations, modality under-utilization, and failures to properly ground predictions in the non-text signal [ 8 , 23 , 24 , 25 ] . Within the audio domain, controlled audio-text disagreement studies provide direct evidence of modality arbitration failures, where models prefer textual instructions even when they directly contradict acoustic ground truths [ 9 , 10 ] . Furthermore, speech affect evaluations show that many LALMs behave more like rigid transcribers than active listeners, failing to disentangle acoustic prosody from lexical content [ 26 ] . Mechanistic Interpretability. Mechanistic interpretability provides tools to localize where and how information is represented and used within transformer computations, linking model behavior to internal mechanisms rather than post-hoc rationales [ 27 ] . A recurring finding is that individual components—especially attention heads—often exhibit specialized and reusable functional roles [ 20 ] , enabling targeted, component-level interventions. More recently, these tools have been extended to multimodal transformers, including analyses that identify modality-linked attention heads and study their causal role across tasks [ 16 , 17 , 28 ] . Within audio and LALM settings, early work applies mechanistic analyses to probe how acoustic evidence propagates through the model and to localize audio-related computations [ 14 ] . A complementary line explores training-free, inference-time interventions that exploit such localization to improve multimodal grounding, e.g., vector-steering methods for audio models [ 29 , 30 ] . A complementary line explores training-free, inference-time interventions that exploit such localization to improve multimodal grounding, including activation steering via adding a learned or contrastive direction to internal representations [ 31 , 32 ] . 3 Preliminaries and Notation Token stream and audio indices. Audio-language transformers operate on a single sequence of n tokens, including both text and audio tokens. We

Read the complete text