SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors

Yekaterina Yegorova, Argyrios Gerogiannis, Haolong Zheng, Julia Hockenmaier, Chang D. Yoo, Mark A. Hasegawa-Johnson · 2026

Read SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors by Yekaterina Yegorova, Argyrios Gerogiannis, Haolong Zheng, Julia Hockenmaier, Chang D. Yoo, Mark A. Hasegawa-Johnson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00460.txt.

Introduction

Yegorova and co-authors adapt speech-aware chat models by learned steer traits in their layers. Such models slip when speech strays from their home field. SALSA learns a steer trait for each layer to shift them. Past steer tricks took traits from paired gap reads.

SALSA trains its traits head-on through a scored task goal. Tests span child speech, world tongues, and mixed Mandarin-English speech. Scores leap past zero-shot and in-chat learn base marks. Top gains reach near half in fair terms. Reads show the sound-side layers steer best, most of all late ones. The shift seems to tune sound traits to the model's own space. Its word side needs no such shift at all.

Catalog record

CREATORYekaterina Yegorova, Argyrios Gerogiannis, Haolong Zheng, Julia Hockenmaier, Chang D. Yoo, Mark A. Hasegawa-Johnson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00460

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors Yekaterina Yegorova, Argyrios Gerogiannis, Haolong Zheng, Julia Hockenmaier, Chang D. Yoo, Mark A. Hasegawa-Johnson Abstract Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight adaptation method that learns layer-wise steering vectors. Unlike commonly used steering approaches that rely on contrastive activation differences, SALSA directly optimizes steering vectors using a supervised objective. Across children’s speech, multilingual speech, and Mandarin-English code-switching benchmarks, SALSA substantially improves performance over zero-shot inference and speech in-context learning baselines, achieving up to 46.8% relative improvements over zero-shot. Analysis further demonstrates that steering the encoder, particularly the later layers, is more effective than steering the LLM backbone. These findings suggest that steering improves downstream ASR performance by adapting higher-level acoustic and phonetic representations to better align with the pretrained language model representation space, rather than by modifying the decoder itself. 1 Introduction Speech-aware large language models (SALLMs) have emerged as a powerful paradigm for speech processing, leveraging linguistic knowledge encoded in pretrained large language models (LLMs) to advance performance across a broad range of speech tasks ( Peng et al. 2025 ) . Despite their strong performance on high-resource languages, they generalize poorly to out-of-domain scenarios, and this occurs even when the underlying model components have already been exposed to the target languages or tasks during pretraining. This suggests that adaptation in SALLMs is not solely a problem of acquiring new linguistic knowledge, but also one of aligning pretrained acoustic representations with the language model decoding space for downstream ASR tasks. Fine-tuning can address these gaps, but is computationally expensive and data-intensive. Parameter-efficient alternatives such as LoRA ( Hu et al. 2021 ) reduce this cost but still require gradient updates to model weights. In-context learning (ICL) ( Brown et al. 2020 ) is training-free, but is constrained by the high acoustic variability of speech. Unlike text, speech encodes speaker identity and phonetic properties alongside semantics, making it difficult to identify informative demonstration examples ( Zheng et al. 2025 ) . A complementary method is activation steering: directly perturbing a model’s internal representations at inference time to shift its behavior toward a target domain, without modifying its weights. Steering has been shown to be effective in language models Im and Li 2025 and has recently started to be explored for speech Sun et al. 2026 ; Feng et al. 2025 . Existing approaches for the speech modality rely on contrastive activation differences, where steering vectors are derived from paired contrastive examples. For automatic speech recognition (ASR), natural paired examples are difficult to obtain, unlike for semantic tasks. ASR requires a direct correspondence between the input audio and output transcript, and the high acoustic variability of speech makes contrastive pairs noisy and hard to construct. This limits the applicability of existing steering methods to exactly the low-resource, out-of-domain settings where efficient adaptation is most needed. In this paper, we propose SALSA ( S peech-aware LLM A daptation via L earned S teering A ctivations), a lightweight adaptation method that directly learns layer-wise steering vectors for speech-language models without requiring paired contrastive examples. By training steering vectors, SALSA can learn around acoustic variability and adapt intermediate encoder representations to better align with the pretrained language model representation space. Our contributions are as follows: • We propose SALSA, a lightweight steering-based adaptation method that learns layer-wise encoder steering vectors for pretrained SALLMs while keeping all backbone parameters frozen. • We demonstrate that simple learned steering vectors are highly effective for out-of-domain ASR settings, including children’s speech, multilingual speech, and code-switched speech, substantially improving over zero-shot and speech in-context learning baselines without requiring paired contrastive examples. • We analyze steering behavior across training scale, encoder depth, and intervention location, showing that adaptation is most effective in higher-level encoder representations, while steering the language model backbone provides substantially smaller gains than encoder steering. 2 Background SALLMs, such as SALMONN ( Tang et al. 2024 ) , WavLLM ( Hu et al. 2024 ) , the Qwen-Audio family ( Chu et al. 2023 ; Chu et al. 2024b ) , and Granite-speech ( Saon et al. 2025 ) , are typically constructed by coupling a pretrained speech encoder with an LLM backbone, where only a lightweight projection layer is trained to bridge the two modalities. These models support a broad range of tasks, from traditional speech processing tasks such as ASR to more open-ended tasks, including conversational speech and question answering. However, relative to speech-native models such as Whisper ( Radford et al. 2023 ) , SALLMs are generally trained on considerably less speech data and tend to focus on a limited set of predominantly European languages. As a result, SALLMs are prone to hallucination and generalize poorly to out-of-domain data. 2.1 Adaptation Methods for SALLMs Adaptation through full fine-tuning or parameter-efficient methods such as LoRA ( Hu et al. 2021 ) can address out-of-domain limitations, but such approaches require modifying model parameters and may be impractical in low-resource or computationally-constrained settings. More recently, reinforcement learning (RL)-based post-training methods have also been explored for improving speech-language models, although these approaches similarly require iterative model updates and large-scale optimization ( Guo et al. 2025 ; Rafailov et al. 2023 ; Elmakies et al. 2026 ) . A lighter-weight alternative is speech in-context learning (SICL) ( Wang et al. 2024 ) , which was originally proposed for Whisper and has since been extended to SALLMs ( Omnilingual et al. 2025 ; Abouelenin et al. 2025 ; Roll et al. 2025 ) . Zheng et al. 2026 further demonstrates that selecting semantically similar in-context examples substantially improves performance across a range of challenging conditions, including children’s speech, accented speech, and multilingual settings. However, unlike text, speech contains substantial variability in speaker identity, prosody, and acoustic conditions, making it difficult to retrieve informative demonstrations across out-of-domain settings. 2.2 Steering Representation steering methods aim to control model behavior by intervening directly in the hidden representations of a neural network during inference. Given a model with hidden representation h_{l}\in\mathbb{R}^{d} at layer l , steering methods modify the forward pass by applying an intervention function \tilde{h}_{l}=f(h_{l},v,\alpha), where v denotes a steering direction or transformation and \alpha controls the intervention strength. The modified representation \tilde{h}_{l} is then propagated through the remainder of the network to influence model outputs. Many steering methods use additive interventions of the form \tilde{h}_{l}=h_{l}+\alpha v, where v corresponds to a direction in representation space associated with a desired behavior or attribute. Early work demonstrated that meaningful behavioral directions can often be extracted from contrastive examples. ( Rimsky et al. 2024 ) computes steering vectors from differences between hidden activations of paired prompts exhibiting opposite behaviors. Given activations h_{l}^{+} and h_{l}^{-} corresponding to positive and negative examples, respectively, a steering vector can be computed as v=\frac{1}{N}\sum_{i=1}^{N}\left(h_{l,i}^{+}-h_{l,i}^{-}\right). This contrastive approach has been extended through PCA-based steering methods ( Liu et al. 2024 ; Zou et al. 2023 ) , which estimate steering vectors from the principal components of contrastive activation differences. By identifying directions that explain the largest variance across contrastive pairs, PCA-based methods aim to isolate more robust and semantically meaningful steering directions. Approaches like Inference-Time

Read the complete text