In-context Language Learning for Endangered Languages in Speech Recognition

Zhaolin, Jan · 2025

Read In-context Language Learning for Endangered Languages in Speech Recognition by Zhaolin, Jan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20445.txt.

Introduction

Zhaolin Li and Jan Niehues wrote a 2025 paper on whether language models can pick up an endangered language just from examples in the prompt. About 7,000 languages are spoken worldwide, and large models support only a small subset. Prior work showed models can learn unseen languages for some text tasks without training data.

The authors extend that question to speech recognition. They tested four diverse endangered languages the models had never been trained on. Feeding more relevant text samples in context improved both language modeling and speech recognition. A probability-based method beat plain instruction prompting. Remarkably, in-context learning let the model match or beat dedicated systems trained for those languages, while keeping its original skills. The code is public. The approach offers endangered-language technology without costly training.

Catalog record

CREATORZhaolin, Jan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20445

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

In-context Language Learning for Endangered Languages in Speech Recognition Zhaolin, Jan Abstract With approximately 7,000 languages spoken worldwide, current large language models (LLMs) support only a small subset. Prior research indicates LLMs can learn new languages for certain tasks without supervised data. We extend this investigation to speech recognition, investigating whether LLMs can learn unseen, low-resource languages through in-context learning (ICL). With experiments on four diverse endangered languages that LLMs have not been trained on, we find that providing more relevant text samples enhances performance in both language modelling and Automatic Speech Recognition (ASR) tasks. Furthermore, we show that the probability-based approach outperforms the traditional instruction-based approach in language learning. Lastly, we show ICL enables LLMs to achieve ASR performance that is comparable to or even surpasses dedicated language models trained specifically for these languages, while preserving the original capabilities of the LLMs. Li Niehues nocounter]Karlsruhe Institute of TechnologyGermany keywords low-resource languages, large language models, speech recognition † † email: [email protected] 1 Introduction State-of-the-art automatic speech recognition (ASR) systems have achieved human-level performance for high-resource languages. However, their performance remains limited for endangered languages [ 1 , 2 , 3 ] . Popular ASR systems incorporate both acoustic and language information. Previous studies have shown that the effectiveness of leveraging acoustic information gets significantly improved for endangered languages, but the integration of language information remains under-explored [ 4 , 5 ] . With the rise of large multilingual language models (LLMs), many such models are now available. Nevertheless, their application in automatic speech recognition (ASR) systems for endangered languages, especially those that are not represented in the training data, remains unclear. In addition to traditional fine-tuning methods, LLMs offer an alternative approach to learning new tasks known as In-Context Learning (ICL). This enables LLMs to perform tasks by conditioning on a few example prompts, without requiring parameter updates [ 6 ] . Instead of fine-tuning, ICL leverages the model’s pre-trained knowledge and adapts it to new tasks. While ICL has demonstrated strong performance for high-resource languages with abundant training data, its effectiveness drops significantly for low-resource languages [ 7 , 8 ] . This decline is primarily due to insufficient linguistic representation in the model’s pretraining data, leading to poor generalization and making ICL less reliable for these languages. To enhance LLM performance on low-resource languages, one research direction is cross-lingual ICL (X-ICL) [ 6 , 9 , 10 ] . X-ICL leverages the model’s multilingual knowledge by providing prompts in a high-resource language or a mix of high- and low-resource languages while expecting the output in the low-resource language. This approach facilitates knowledge transfer across linguistic boundaries and mitigates data scarcity. However, X-ICL often introduces translation biases and struggles with structural inconsistencies between languages. Additionally, its performance tends to decline in extremely low-resource scenarios, particularly for languages the model has never encountered. Another research direction is long-context ICL, which extends the context length available for in-context learning, enabling models to process more examples and capture richer patterns [ 11 , 12 ] . By incorporating longer prompts with more diversity, long-context ICL can improve generalization to low-resource languages. However, it comes with challenges such as increased computational cost, sensitivity to input ordering, and diminishing returns when context length exceeds the model’s optimal processing capacity. Language ISO code Language Family Audio source Train samples Train (h) Dev+Test (h) Khinalug kjj Northeast Caucasian Spontaneous 978 2.14 0.49 Kichwa que Quechuan Radio 2991 3.05 0.77 Mboshi mdw Bantu ZoneC Reading 4513 3.93 0.53 Japhug jya Sino-Tibetan Spontaneous 23975 27.74 7.00 Table 1: Dataset descriptive statistic Beyond exploring approaches to boost ICL performance on low-resource languages, researchers have investigated different evaluation strategies to assess ICL performance across various tasks. Common evaluation benchmarks for low-resource languages include machine translation [ 13 ] , which tests cross-lingual transferability; code-switching [ 14 ] , which evaluates the model’s ability to handle mixed-language inputs; interlinear text glossing [ 12 ] , which examines morphological and syntactic understanding. However, the Automatic Speech Recognition (ASR) task remains significantly underexplored. Unlike text-based tasks, ASR faces unique challenges, such as the acoustic variability caused by diverse accents and speaker demographics, the complex phonetic and prosodic features that are often poorly represented in pretrained models. Moreover, low-resource languages sometimes lack standardized orthography, making ASR even more difficult compared to purely tasks-based tasks. Despite these challenges, some studies have attempted to apply ICL in ASR for low-resource languages. [ 15 ] explored ICL by feeding relevant audio-text pairs alongside the target audio into ASR models. However, long audio inputs restrict the number of ICL samples that can be used, limiting overall performance. Additionally, their approach required LLMs that support audio input, further narrowing the choice of models. Motivated by previous research, we propose not only learning a new task through ICL but also a new language. By In-context language learning (ICLL), LLMs can leverage their knowledge of multiple languages to learn an entirely new language using just a few hundred sample sentences. We evaluate this capability in the context of ASR, assessing how well LLMs can adapt to new languages for this task. Additionally, ICLL offers the advantage of selecting different samples for each test instance, allowing better adaptation to individual text examples. To optimize performance, we investigate various sample selection strategies. We summarize our main contributions below. • ICLL enables LLMs to learn a new language with only a few hundred samples. • ICLL achieves better or comparable performance to corpus-trained language models in language modelling and ASR tasks. • Sample selection in ICLL is important to achieve strong performance. 2 In-context Language Learning This work aims to investigate language modelling with ICL for low-resource languages, which means the objective is not to teach a new task to LLMs through examples within the prompt, but to provide linguistic knowledge of the target language to LLMs. Even though, the general setup for both objectives remains the same. The setup begins with prompt creation, which consists of sentences in the target language without additional instructions. Due to computational resources constraints, it is impractical to feed all available text within the prompt, even in the scenarios of low-resource languages. To address this, we investigate various sample selection strategies with explanations in Section 2.1 . Once the prompt is provided, LLM generates text in the target language with task-specific requirements to perform ICL. However, in the use case of ASR, this work focuses on not generating text but on ranking different possible transcriptions. To achieve this, we use an initial ASR system to generate an n-best list of candidate transcriptions, which is subsequently re-ranked using ICL. We explore different methods for selecting the best matching hypothesis to optimize this process, as explained in Section 2.2 . 2.1 Samples Retrieval Strategy The baseline approach is to select random samples. As the most general method for learning the language, this may not lead to the best performance since the samples do not align with the specific task [ 12 , 15 ] . In traditional language modelling, corpus-level selection is typically performed, where a single set of samples is selected for the test set. This is because training a language model requires substantial computation. However, the drawback is

Read the complete text