Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages

Seraphina, Marco, Alessio · 2025

Read Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages by Seraphina, Marco, Alessio (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05149.txt.

Introduction

Seraphina Fong, Marco Matassoni, and Alessio Brutti ask how much speech data a data-poor AI needs to hear words right. Top speech models shine in rich languages, yet most languages of the world have thin data. Their 2025 study uses a frame where a light bridge links a speech reader to a language model. It grades two multilingual language models on known speech test sets.

First, the team counts the data volume it takes to tie a strong speech-only tool. That volume is steep, which backs the fear that thin languages start far back. Next, bridges first trained on rich languages cut the pain a lot, most of all when the new set is tiny. Languages with one bridge or many both helped. The readings give a scale for low-data speech work.

Catalog record

CREATORSeraphina, Marco, Alessio
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05149

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages Seraphina, Marco, Alessio Abstract Large language models (LLMs) have demonstrated potential in handling spoken inputs for high-resource languages, reaching state-of-the-art performance in various tasks. However, their applicability is still less explored in low-resource settings. This work investigates the use of Speech LLMs for low-resource Automatic Speech Recognition using the SLAM-ASR framework, where a trainable lightweight projector connects a speech encoder and a LLM. Firstly, we assess training data volume requirements to match Whisper-only performance, re-emphasizing the challenges of limited data. Secondly, we show that leveraging mono- or multilingual projectors pretrained on high-resource languages reduces the impact of data scarcity, especially with small training sets. Using multilingual LLMs (EuroLLM, Salamandra) with whisper-large-v3-turbo, we evaluate performance on several public benchmarks, providing insights for future research on optimizing Speech LLMs for low-resource languages and multilinguality. Fong Matassoni Brutti University of TrentoItaly Fondazione Bruno KesslerItaly keywords speech recognition, LLM, low-resource languages, multilinguality † † email: [email protected], [email protected], [email protected] 1 Introduction Low-resource settings, characterized by limited data or lack of manual annotations, remain a major challenge for speech technologies in production environments [ 1 ] . For example, current Automatic Speech Recognition (ASR) models are limited by the amount and diversity of training data required, supporting only about 100 of the more than 7,000 languages spoken globally [ 2 , 3 , 4 ] . As a result, the scarcity of high-quality training data continues to hinder advances in speech and language technologies for low-resource scenarios [ 3 , 4 , 5 , 6 ] . Therefore, the need for technologies that support underrepresented languages is increasingly important, to not only preserve linguistic and cultural heritage, but to also ensure that a broader range of users worldwide have access to modern technologies. Originally developed for text generation, large language models (LLM) have demonstrated increasing versatility in performing an expanding range of tasks [ 7 , 8 , 9 , 10 ] . Of particular interest to the present work is how they have been extended to process speech and audio, enabling new possibilities for speech-related tasks. Recent LLM-based ASR approaches have implemented a trainable projector to integrate speech encoders with LLMs to improve ASR performance by leveraging the extensive knowledge of LLMs [ 11 , 12 , 13 , 14 , 15 ] . The projector’s role is to align speech embeddings from the speech foundation encoder with the LLM’s embedding space, enabling an LLM to handle speech input alongside textual prompts. The SLAM-ASR [ 16 ] approach demonstrated that implementing a lightweight layer as the projector between off-the-shelf speech foundation encoders and LLMs could achieve state-of-the-art results on the English Librispeech dataset [ 17 ] . However, most emerging research in this domain predominantly focuses on high-resource languages (e.g., English, Mandarin Chinese), and therefore have large amounts of data available for training (e.g., 960 hours of Librispeech for SLAM-ASR [ 16 , 18 ] , 1000 hours of data per language in [ 15 ] ). Other studies have investigated the impact of factors such as data characteristics or computational resources [ 18 ] . In particular, [ 19 ] demonstrates the crucial role of training data volumes for LLM-based audio-visual ASR. However, it remains unclear how well the LLM-based framework performs for low-resource settings, as data scarcity can impact all components of the pipeline. For example, the speech encoder may learn less robust representations, the LLM may generate less accurate transcripts due to limited exposure to the target language, and the projector may struggle to effectively align the different modality embeddings. Furthermore, in low-resource settings, there is often little to no data available not only for ASR, but also for other speech-related tasks such as spoken language understanding and spoken question answering. Given these challenges, it is essential to investigate the viability of Speech LLMs in low-resource scenarios. In this work, we use ASR as a case study. Therefore, our research questions (RQ) are as follows: ( \mathbf{RQ1} ) How many hours of training data are needed to effectively train a linear projector? ( \mathbf{RQ2} ) Can linear projectors pretrained on a high-resource language be fine-tuned on another language to mitigate the impact of data scarcity in low-resource ASR? To address these questions, we progressively increase the amount of data used to train a linear projector within the SLAM-ASR framework, starting from 10 to 252 hours of the Common Voice (CV) Italian dataset [ 20 ] , to determine the amount of data needed to match or exceed the performance of using a Whisper-only framework [ 21 ] . We subsequently explore the effects of leveraging a projector pretrained on English data (Librispeech 100 [ 17 ] and 100 hours of CV English) or 200 hours of CV Spanish data, fine-tuning them on varying amounts of Common Voice Italian data (10, 15, 100, and 200 hours) to assess its potential adaptability and performance in low-resource settings. We further extend this analysis to a real-world low-to-medium resource scenario with 10-15 hours of Galician data, where we also bootstrap a multilingual projector pretrained with a combination of English, Spanish, and Italian data. Consequently, we also investigate the impact of the selected LLM using two open-source multilingual LLMs (EuroLLM 1.7B [ 22 ] and Salamandra 2B [ 23 ] ), as well as reinforce previous findings [ 18 ] of SLAM-ASR performance in cross-domain evaluation settings. Overall, this work presents the following contributions to provide insights into the applicability of Speech LLMs in low-resource scenarios, using SLAM-ASR as a case-study framework: • Achieving performance comparable to Whisper-only models requires 200 hours of training data, highlighting the persistent challenge of data scarcity in low-resource settings. • Fine-tuning projectors pretrained on high-resource languages enhances cross-domain performance, especially when the fine-tuning dataset is limited (10–15 hours). Performance is further improved when leveraging projectors pretrained on multiple languages (i.e., a “multilingual” projector). • The choice of multilingual LLM impacts performance, with EuroLLM 1.7B performing better than Salamandra 2B. 2 Methodology We employ the popular SLAM-ASR framework and code-base for our experimental analysis [ 16 ] , chosen for its open-source accessibility and strong performance on English benchmarks. As depicted in Figure 1 , SLAM-ASR consists of a frozen speech foundation encoder, a trainable linear projector, and a frozen LLM. The input speech signal, its corresponding transcript, and the text prompt are denoted as X_{s} , X_{T} , and X_{P} , respectively. For each input sample X_{s} , speech embeddings H_{s} are first extracted from the speech signal X_{s} via the speech encoder where: H_{s}=Encoder(X_{s}) (1) We consider the speech encoder from the Whisper-large-v3-turbo model 1 1 1 https://huggingface.co/openai/whisper-large-v3-turbo [ 21 ] , a state-of-the-art transformer-based system for multilingual speech recognition. The extracted speech embeddings are downsampled by a factor of k = 5 to address the length discrepancy between speech and text features. A linear projector then transforms the downsampled H_{s} into E_{s} to align with the LLM’s input embedding dimension. Following the original SLAM-ASR implementation, the linear projector consists of a single hidden layer followed by ReLU activation and a regression layer, denoted as: E_{s}=Linear(ReLU(Linear(Downsample(H_{s}))) (2) The transcript X_{T} and prompt X_{P} are

Read the complete text