Activation Steering for Accent Adaptation in Speech Foundation Models
Jinuo, Yang, Sung Kyun, Qiuchi, Gongping, Eun-Jung, Ting · 2026
Read Activation Steering for Accent Adaptation in Speech Foundation Models by Jinuo, Yang, Sung Kyun, Qiuchi, Gongping, Eun-Jung, Ting (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05813.txt.
Introduction
Sun and colleagues treat accent variation as an interpretable subspace inside hidden representations of speech recognition models. Most adaptation methods fine-tune parameters without locating where accent information lives. The authors extract encoder activations layer by layer and estimate mean-shift directions for accent-induced shifts. Injecting those directions layer by layer shows accent information clustering in middle encoder layers. Building on that structure, they introduce parameter-free steering that reshapes representations during inference without updating weights. Tests across eight accents show steady word error rate cuts.
Catalog record
| CREATOR | Jinuo, Yang, Sung Kyun, Qiuchi, Gongping, Eun-Jung, Ting |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25687 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05813
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Activation Steering for Accent Adaptation in Speech Foundation Models Jinuo, Yang, Sung Kyun, Qiuchi, Gongping, Eun-Jung, Ting Abstract Accent variability remains a major errors in automatic speech recognition, yet most adaptation methods rely on parameter fine-tuning without understanding where accent information is encoded. We treat accent variation as an interpretable subspace in hidden representations and investigate whether it can be identified and controlled directly in activation space. We extract layer-wise encoder activations and estimate mean-shift directions capturing accent-induced representation shifts. By injecting these directions into individual layers and measuring how they align accented and standard embeddings, we derive a layer-wise accent sensitivity profile, revealing that accent information concentrates in a narrow band of middle encoder layers. Leveraging this structure, we further introduce parameter-free accent steering that modifies representations during inference without updating model weights. Experiments across eight accents show consistent word error rate reductions. Sun Xiao Chung Hu Huang Holden Dang keywords speech recognition, human-computer interaction, computational paralinguistics † † address: 1 School of Computer Science, The University of Melbourne, Australia 2 Wuhan University, China † † email: [email protected], [email protected], [email protected] 1 Introduction Accent variability remains a persistent challenge for automatic speech recognition (ASR) systems [ 13 ] . Systematic differences in phoneme realization, prosody, and phonotactic patterns across regional and non-native accents often lead to recognition errors that disproportionately affect certain speaker populations [ 29 , 31 ] . In real-world deployments, including voice assistants, call centers, and educational technologies, such performance disparities degrade user experience and raise important concerns regarding fairness and accessibility [ 9 ] . Despite advances in large-scale pretraining and robust acoustic modeling [ 27 ] , recognition gaps across accents continue to be observed. Conventional accent adaptation techniques typically rely on supervised fine-tuning, accent-specific modeling, or data augmentation [ 4 , 19 , 16 ] . While effective in traditional ASR settings, these approaches become increasingly costly and operationally restrictive in the era of foundation-scale speech models. Full-parameter adaptation is computationally expensive and may compromise generalization across diverse accents and tasks [ 23 , 8 , 11 ] . As modern speech systems are frequently deployed as shared, large-scale backbones, there is growing need for lightweight and scalable adaptation mechanisms. Recent large audio language models (LALMs), such as Whisper [ 20 ] , combine powerful acoustic modeling with large language modeling to achieve strong zero-shot generalization. While Parameter-Efficient Fine-Tuning (PEFT) methods and bottleneck adapters [ 5 , 6 , 18 , 30 ] offer a more lightweight adaptation approach by adding only a small number of trainable parameters on top of a frozen base model, these approaches still optimize the added parameters heuristically [ 22 , 17 , 1 , 28 ] . Without explicitly localizing or constraining updates to layers and subspaces that are most sensitive to accent-related variation, these methods may perform unnecessary adaptation in accent-insensitive regions of the model and risk entangling accent compensation with higher-level semantic representations, limiting both efficiency and control. Understanding how accent variation is organized in representation space is critical for designing scalable and controllable adaptation strategies. If accent corresponds to identifiable subspaces in hidden activations, then targeted representation-level interventions may offer an alternative to parameter-intensive fine-tuning [ 26 ] . Recent studies in large language models show that high-level attributes, such as sentiment, style, or safety [ 2 , 12 , 25 , 24 , 32 ] , can align with approximately linear directions in activation space. These directions, often referred to as steering vectors, are typically constructed as mean activation shifts between contrasting concepts [ 21 , 7 ] . When added to hidden states at selected layers, they nudge representations toward desired regions of the space, enabling controllable behavioral modulation at inference time without modifying model parameters. This perspective motivates the central question of this work: Does accent variation correspond to a structured and controllable subspace in LALMs? In this paper, we first conduct a layer-wise analysis of hidden activations to understand the geometric structure of accent variations. This investigation serves two primary purposes. From an interpretability perspective, it reveals how the model distributes accent information across layers. Furthermore, from a controllability perspective, it determines whether these accent features are organized enough to allow direct adjustments. Based on this foundation, we construct steering vectors to adjust accent representations during model inference. Our method inserts learned accent directions into specific hidden states. We subsequently evaluate their effectiveness in automatic speech recognition (ASR) tasks across diverse pronunciations. To the best of our knowledge, this is the first study to systematically analyze and apply vector steering for accent-robust automatic speech recognition. Our extensive experiments on the VCTK and L2-ARCTIC datasets demonstrate the middle layers and late layers are the most sensitive to accent-induced representation shifts. Moreover, mean shift steering significantly improves the word error rate across eight distinct accents. We observe that interventions in the middle layers provide the most effective control. This study offers a principled and scalable way to reduce accent-induced recognition disparities in speech foundation models, advancing both robustness and fairness in real-world ASR deployment. 2 Accent Subspace Analysis and Steering 2.1 Layer-wise Accent Subspace Analysis We analyze how accent-related variation is organized across encoder layers and quantify how strongly each layer responds to accent-specific representation shifts. This yields a layer-wise sensitivity profile that highlights which layers are most suitable for subsequent accent steering. 2.1.1 Accent Representation Perturbation We construct text-matched utterance pairs to isolate accent-related acoustic variation from linguistic content. For each target accent, we create two types of pairs: (i) cross-standard-accent pairs (x_{s},x_{a}) , where x_{a} is an accented utterance and x_{s} is a standard-English utterance with an identical transcript, and (ii) within-single-accent pairs (x_{a},x_{a}) formed by different speakers from the same accent group. The cross-standard-accent pairs capture systematic accent-induced differences, while the within-single-accent pairs serve as a control to account for speaker-specific factors such as timbre or prosody, helping us separate accent effects from general inter-speaker variation. Given accented speech samples {x_{a}} and standard speech samples {x_{s}} , our layer-wise analysis then quantifies, at each encoder layer, how the latent representations of accented speech differ from those of standard English, and how effectively a layer supports aligning these two clusters. Concretely, for each layer l , we first extract the token-level hidden activations \mathbf{H}^{(l)} . For each speech sample i , we apply mean pooling over time to obtain an utterance-level representation \bar{\mathbf{h}}_{i}^{(l)} . Based on these pooled representations, we compute an accent mean-shift direction from a source accent group s to a target accent group a : \mathbf{d}_{s\rightarrow a}^{(l)}=\frac{1}{|G_{s}|}\sum_{j\in G_{s}}\bar{\mathbf{h}}_{j}^{(l)}-\frac{1}{|G_{a}|}\sum_{i\in G_{a}}\bar{\mathbf{h}}_{i}^{(l)} (1) where G_{s} and G_{a} denote the sample sets corresponding to the standard and accent speech, respectively. This vector \mathbf{d}_{s\rightarrow a}^{(l)} characterizes the accent shift between the two representation clusters at layer l . To probe the sensitivity of layer l to this accent direction, we apply a controlled perturbation to its hidden activations: \tilde{\mathbf{H}}^{(l)}=\mathbf{H}^{(l)}+\alpha\cdot\mathbf{d}_{s\rightarrow a}^{(l)}, (2) where \alpha is optimized to 1.0 , corresponding to a one-unit accent mean-shift perturbation, and \mathbf{d}_{s\rightarrow a}^{(l)} is broadcast across all time steps of \mathbf{H}^{(l)} . The resulting perturbed activations \tilde{\mathbf{H}}^{(l)} are then propagated through the remaining encoder and projector layers. 2.1.2 Quantifying Layer Sensitivity via Accent Alignment Score (AAS) Given the