Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages Yang Xiao , Eun-Jung Holden , Ting Dang The University of Melbourne Abstract Recent speech foundation models excel at multilingual automatic speech recognition (ASR) for high-resource languages, but adapting them to low-resource languages remains challenging due to data scarcity and efficiency constraints. Full-model fine-tuning is computationally expensive and prone to overfitting, while parameter-efficient methods like LoRA apply adaptation uniformly across layers, overlooking internal representations thus compromising effectiveness and efficiency. We analyze multilingual ASR models and reveal a U-shaped adaptability pattern: early and late layers are language-specific and require more adaptation, while intermediate layers retain shared semantics and need less. Building on this observation, we propose DAMA, a Depth-Aware Model Adaptation framework that allocates adaptation capacity according to each layer’s role. DAMA also introduces Singular Value Decomposition (SVD)-based initialization to constrain adaptation and preserve the U-shaped pattern, as well as a frozen middle-layer basis for further efficiency. Evaluated on 18 low-resource languages across two benchmark datasets, DAMA matches or surpasses state-of-the-art accuracy with 80% fewer trainable parameters, achieves a 29% error reduction under extreme data scarcity, and significantly improves memory, training time, and computational efficiency over baselines. These results highlight the benefits of structure-aware adaptation for efficient, scalable multilingual ASR. 1 Introduction Recent speech foundation models ( 20 ; 5 ) are pretrained on vast amounts of multilingual speech data and capable of performing a variety of tasks, such as multilingual automatic speech recognition (ASR), with particular success in high-resource languages such as English. However, their performance drops substantially on low-resource languages or local dialects ( 22 ) , due to the limited data availability, resulting in substantial disparities for underrepresented languages. Efficiently adapting these models for low-resource languages with limited data remains a significant challenge. In many real-world scenarios, such as the rapid deployment of speech technologies in emerging markets or for local dialects, there is a critical need for adaptation methods that are both accurate and highly efficient. These approaches must enable fast adaptation to limited new data, operate within stringent memory and computational constraints, and maintain strong performance even in extremely low-data settings. Traditional full-parameter fine-tuning is computationally and memory intensive, making it impractical for many applications, especially on edge devices or with limited resources. The scarcity of labeled data for low-resource languages further complicates adaptation, as full-parameter fine-tuning often suffer from overfitting or catastrophic forgetting ( 25 ; 3 ; 32 ; 12 ) . Recent work has shifted towards parameter-efficient fine-tuning (PEFT) ( 7 ) , such as Low-Rank Adaptation (LoRA) ( 9 ) , which freezes the original model parameters and updates only a small set of new weights for the target language. These methods enable faster and more memory-efficient adaptation. However, these methods adapt all layers uniformly in a brute-force manner, overlooking the structure of language representations and potentially limiting effectiveness and parameter efficiency, especially in extremely low-resource settings. In large models, even standard LoRA may still require updating a significant number of parameters. Therefore, advancing efficient multilingual adaptation in speech foundation models requires a nuanced understanding of how these models represent and share multilingual knowledge internally. This study first conducts a layer-wise analysis of how multilingual speech representations are maintained and interact across different model layers, revealing a distinct U-shaped pattern of plasticity: early and late layers capture language-specific features and are more adaptable, while middle layers remain language-agnostic. This suggests that different layers require varying degrees of adaptation to new languages, challenging the prevailing assumption in prior work that all layers are equally suitable for adaptation ( 23 ; 11 ) . Motivated by this U-shaped pattern, we propose Depth-Aware Model Adaptation (DAMA), which introduces three mechanisms to tune models for new languages with both effectiveness and efficiency. First, the Depth-Aware Rank Schedule allocates higher adaptation capacity to the more plastic early and late layers while restricting the rank in the middle layers, balancing parameter efficiency with preservation of the model structural properties. Second, to constrain adaptation in the middle layers, we propose SVD-Based Initialization, which initializes adaptation weights in directions orthogonal to the dominant weights of the model. This helps preserve shared language representations and maintain the U-shaped adaptability. Finally, to further improve efficiency, especially in low-resource settings, we introduce Basis-Protected Projection (BPP), where a subset of adaptation weights is frozen, thus reducing the number of trainable parameters while preserving essential knowledge. We evaluated DAMA on 18 low-resource languages using the Common Voice and FLEURS datasets. DAMA matches or outperforms state-of-the-art baselines while reducing parameters by about 80%. More importantly, in extremely low-resource settings (0.5 to 1 hour of data), it achieves up to 29% relative Word Error Rate (WER) improvement. Efficiency analysis shows a 24% gain in GPU memory utilization and 36% faster training. These results highlight that adaptation aligned with model layer properties enables scalable, parameter-efficient multilingual systems without sacrificing accuracy. Our contribution is summarized below: • We are the first to systematically analyze layer wise multilingual language representations in speech foundation models. We reveal a U-shaped distribution of language specificity, which demonstrates how these models maintain and share cross-lingual knowledge. • We propose DAMA, a novel depth-aware multilingual ASR adaptation framework that achieves an effective balance between adaptability and parameter efficiency, while preserving essential multilingual knowledge. • Our experiments on 18 languages demonstrate a Pareto-optimal trade-off for the proposed DAMA. It exceeds or matches SOTA performance while significantly reducing trainable parameters, memory usage and training time. Most importantly, DAMA exhibits superior robustness in low-resource settings. 2 Related Work Multilingual Automatic Speech Recognition (MASR). MASR ( 30 ) aims to transcribe speech across diverse languages using a single, unified foundation model. Recent advancements have been driven by scaling up training data and model capacity, such as Whisper ( 21 ) and MMS ( 20 ) . While these models excel at recognizing high-resource languages, their performance degrades significantly when applied to low-resource languages unseen during pre-training. Due to the limited data availability, it necessitates the efficient adaptation to new languages without the prohibitive computational cost of full retraining or the risk of catastrophic forgetting ( 13 ) . Efficient Multilingual Adaptation Strategies. To adapt speech foundation models to new languages, fully fine-tuning has traditionally been the default approach. However, updating all parameters is computationally prohibitive for resource-constrained settings. More importantly, FFT often leads to catastrophic overfitting where the model memorizes sparse data at the expense of generalizable semantic knowledge ( 28 ; 32 ; 18 ) . Some methods attempt to preserve prior knowledge by constraining updates to important parameters ( 29 ; 27 ) . However, these methods often struggle to balance the plasticity-stability dilemma, limiting their ability to learn new tasks effectively. To enable efficient adaptation, PEFT has become the main paradigm. Methods such as Low-Rank Adaptation (LoRA) ( 9 ) and Adapters ( 8 ) freeze the pre-trained backbone and inject a small number of trainable parameters to capture task-specific shifts. Despite their success, standard PEFT methods in MASR overlook language representations and adapt all model layers uniformly, compromising both efficiency and effectiveness ( 23 ; 14 ; 31 ) . While methods like AdaLoRA ( 34 ) introduce dynamic rank allocation, they depend on sensitivity scores derived from training data. In few-shot or low-resource settings, insufficient data renders these sensitivity estimates unstable because they rely on computationally expensive searches. This highlights a critical gap in more effective and efficient adaptation mechanisms for low-resource speech data. 3 Analysis of Language Representations To investigate how language representations are maintained and shared within the latent space, we conduct a layer-wise analysis utilizing linear probing at each layer to perform a language identification (LID) task. If the representations at a given layer are language-specific, the LID accuracy will be high; conversely, lower accuracy suggests more language-agnostic representations. This analysis enables us to identify the extent to which language-specific information is preserved across layers. 3.1 Layer-wise Linear Probing Specifically, we conduct our analysis within the encoder-decoder framework, which is commonly used in recent speech foundation models ( 21 ; 17 ; 19 ) . Given a speech input sequence x_{t} , the encoder processes this input and generates a sequence of latent representations, denoted as \mathbf{h}_{t}=\mathrm{Encoder}(x_{t}) . These encoder outputs \mathbf{h}_{t} are then provided as input to the decoder, f_{\theta} , parameterized by weights \theta . The decoder consists of L layers, with the intermediate activations at the l -th layer denoted as \mathbf{z}_{t}^{(l)} , where l\in\{1,\dots,L\} . For each decoder layer l , we apply linear probing to the representations \mathbf{z}_{t}^{(l)} . Specifically, we train a linear classifier g^{(l)}(\cdot) on the activations \mathbf{z}_{t}^{(l)} to perform the LID task with cross-entropy loss. The classification accuracy provides a quantitative measure of the degree to which language-specific information is encoded at the l -th layer. This provides a comprehensive view of how language representations are maintained, diminished, or abstracted at different depths of the decoder network. Dataset and Language Selection: To ensure the robustness of our analysis, we select a diverse set of languages from the Common Voice ( 1 ) dataset. We construct two distinct evaluation groups to test the generalization capability of the model. The first group consists of five languages that were seen during the pre-training, which allows us to measure how the model represents known knowledge. The second group consists of ten unseen languages, which allows us to observe how the model handles completely new linguistic patterns. We provide the full list of these languages and their specific details in Appendix A. (a) Linear Probing Accuracy of Five Seen Languages. (b) Linear Probing Accuracy of Ten Unseen Languages. (c) Linear Probing Accuracy of Five Seen Languages after Fine-tuning. (d) WER after fine-tuning on English. Figure 1: Layer-wise probing for different languages before and after fine-tuning. 3.2 The U-Shaped Layer-wise Plasticity Analysis of Seen Languages: The results on seen languages, as illustrated in Figure 1(a) , reveal a distinct U-shaped pattern across the depth of the decoder. The early layers (layers 1 to 5) and the late layers (layers 28 to 32) achieve a near 100%. This indicates that the model retains strong language-specific markers. Conversely, the intermediate layers exhibit a noticeable drop in performance. Specifically, the accuracy falls to approximately 93% around layer 17. The relative decline forms a “ Semantic Valley ”, which suggests that the middle layers are less sensitive to language identity and focus more on language-agnostic semantic representations. Analysis of Unseen Languages: Further analysis of languages that were not seen during pre-training also reveals the same “Semantic Valley ”, despite the decreased accuracy. This consistent U-shaped behavior confirms that decoder naturally organizes information with depth-dependent plasticity. Regardless of the languages, the early and late layers capture language-specific linguistics features, while the middle layers relatively maintain a language-agnostic representation. It is interesting to note that this U-shaped pattern aligns with recent findings in text-based Large Language Models ( 10 ; 24 ; 26 ) . Our results demonstrate that this hierarchical phenomenon is also evident in speech foundation models, even with the different end-to-end learning paradigm of speech training. 3.3 The Impact of Fine-tuning We further investigate how the commonly used fine-tuning affects the U-shaped plasticity. We first perform full parameter fine-tuning on two distinct languages, Turkish (tr) and Mandarin (zh), and repeat the probing analysis. The results, presented in Figure 1(c) , shows that the distinct U-shaped pattern has completely disappeared, where the “ Semantic Valley ” observed in the pre-trained model is disrupted by a continuous upward trend. The middle layers, which captures language-agnostic representations, have been forced to encode strong language-specific information. While it enables adaptation to new languages, the semantic collapse coincides with catastrophic forgetting of seen languages such as English, as shown in Figure 1(d) , where the WER on English surges from 11% to over 93% after fine-tuning. Further, it also results in significant computational costs, including increased memory footprint and longer training times. While standard LoRA improves adaptation efficiency and preserves the original model weights, it does not account for the specialized role of the middle layers. By uniformly adapting all layers, including those responsible for language-agnostic processing, LoRA may still disrupt critical semantic representations, eroding well-established structures and undermining cross-lingual generalization and robustness. Preserving the U-shaped plasticity, particularly in the middle layers, is essential to safeguard learned knowledge and improve efficiency, as adaptation should primarily target the early and late layers. Figure 2: Overview of the DAMA Framework compared with LoRA. (a) The standard LoRA with uniform rank. (b) The DAMA Framework. Specifically, the Depth-Aware Rank schedule allocates high plasticity to the early and late layers, while the Basis-Protected Projection physically locks the middle layers to protect “Semantic Valley”. All layers mean all the layers from the decoder. The decoder processes acoustic embeddings from the encoder and transcribes them into output tokens. 4 Proposed DAMA To maintain the U-shaped structure during adaptation, we propose the efficient Depth-Aware Model Adaptation (DAMA). First, We introduce a Depth-Aware Rank Schedule assigning higher adaptation capacity to the early and late layers while restricting the lower adaptation in the middle layers. Second, we propose SVD-based Initialization for LoRA in the middle layers to explicitly preserve language-agnostic representations, enabling efficient low-rank adaptation. Third, we design Basis-Protected Projection (BPP), which protects and freezes parameters in the middle layers to further improve adaptation efficiency and stability, especially in low-resource settings. 4.1 Depth-Aware Rank Allocation Standard LoRA ( 9 ) adapts pre-trained models by fine-tuning a low-rank update to each weight matrix. Given a pre-trained model with weight matrix W_{0}\in\mathbb{R}^{d\times k} , LoRA constrains the update \Delta W as the product of two low-rank matrices: B\in\mathbb{R}^{d\times r} and A\in\mathbb{R}^{r\times k} , where r\ll d,k . The forward pass is then: h=(W_{0}+\Delta W)x=W_{0}x+BAx, (1) During training, W_{0} is frozen. A is randomly initialized and B is set to zero, so initially, the update has no effect. Standard LoRA assigns the same rank r to all layers, allocating uniform adaptation capacity to early, middle, and late layers. Instead, DAMA employs the Depth-Aware Rank scheduling following the U-shaped distribution, with a layer-dependent rank function r(l) that flexibly adjusts the adaptation capacity based on the position l of each decoder layer. Specifically, we divide the L decoder layers into three segments: Early ( l_{\text{early}}:1\leq l\leq l_{\mathrm{early}} ), Mid ( l_{\text{mid}}:l_{\mathrm{early}}