Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection
Sergio Burdisso 1 , ⋆ {}^{1,^{\star}} , Esaú Villatoro-Tello 1 , ⋆ {}^{1,^{\star}} , Shashi Kumar 1,2 , Srikanth Madikeri 4 , Andrés Carofilis 1 Pradeep Rangappa 1 , Manjunath K E 3 , Kadri Hacioglu 3 , Petr Motlicek 1,5 , Andreas Stolcke 3 · 2026
Read Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection by Sergio Burdisso 1 , ⋆ {}^{1,^{\star}} , Esaú Villatoro-Tello 1 , ⋆ {}^{1,^{\star}} , Shashi Kumar 1,2 , Srikanth Madikeri 4 , Andrés Carofilis 1 Pradeep Rangappa 1 , Manjunath K E 3 , Kadri Hacioglu 3 , Petr Motlicek 1,5 , Andreas Stolcke 3 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20898.txt.
Introduction
Sergio Burdisso and a large team of co-authors make speech recognition less touchy about its prompt. Their 2026 paper concerns LLM-based automatic speech recognition. In this design, a speech foundation model feeds a large language model. A projector links the two parts. The approach yields promising results. Yet most systems use one fixed, hand-written prompt at training and test time.
That fixed prompt is a weak point. Small wording changes can sway the model's output. The authors add a learnable projection to cut this prompt sensitivity. Learning the prompt's role, rather than freezing text, steadies the system. The work helps voice tools that must work for many speakers and phrasings. It is a small change with a wide effect on reliability.
Catalog record
| CREATOR | Sergio Burdisso 1 , ⋆ {}^{1,^{\star}} , Esaú Villatoro-Tello 1 , ⋆ {}^{1,^{\star}} , Shashi Kumar 1,2 , Srikanth Madikeri 4 , Andrés Carofilis 1 Pradeep Rangappa 1 , Manjunath K E 3 , Kadri Hacioglu 3 , Petr Motlicek 1,5 , Andreas Stolcke 3 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20898
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection Sergio Burdisso 1 , ⋆ {}^{1,^{\star}} , Esaú Villatoro-Tello 1 , ⋆ {}^{1,^{\star}} , Shashi Kumar 1,2 , Srikanth Madikeri 4 , Andrés Carofilis 1 Pradeep Rangappa 1 , Manjunath K E 3 , Kadri Hacioglu 3 , Petr Motlicek 1,5 , Andreas Stolcke 3 Abstract LLM-based automatic speech recognition (ASR), a well-established approach, connects speech foundation models to large language models (LLMs) through a speech-to-LLM projector, yielding promising results. A common design choice in these architectures is the use of a fixed, manually defined prompt during both training and inference. This setup not only enables applicability across a range of practical scenarios, but also helps maximize model performance. However, the impact of prompt design remains underexplored. This paper presents a comprehensive analysis of commonly used prompts across diverse datasets, showing that prompt choice significantly affects ASR performance and introduces instability, with no single prompt performing best across all cases. Inspired by the speech-to-LLM projector, we propose a prompt projector module, a simple, model-agnostic extension that learns to project prompt embeddings to more effective regions of the LLM input space, without modifying the underlying LLM-based ASR model. Experiments on four datasets show that the addition of a prompt projector consistently improves performance, reduces variability, and outperforms the best manually selected prompts. † † address: 1 Idiap Research Institute 2 EPFL 3 Uniphore 4 University of Zurich 5 Brno University of Technology 1 Introduction † † footnotetext: ⋆ Corresponding authors: { sergio.burdisso, esau.villatoro } @idiap.ch Integrating speech capabilities into large language models (LLMs) is a key research area, enabling seamless voice interaction and enhancing multimodal understanding [ 1 , 2 , 3 , 4 ] . A conventional approach uses a cascaded architecture, first with an automatic speech recognition (ASR) system followed by an LLM to process the recognized text [ 5 , 6 , 7 , 8 ] . However, this pipeline suffers from error propagation and prevent the LLM from having access to prosodic information that could be relevant for downstream tasks. A promising alternative is LLM-based ASR, which directly connects a speech foundation model to an instruction-tuned LLM via a lightweight speech projector [ 9 , 10 , 3 ] . This projector maps speech-derived embeddings into the LLM’s input space, enabling direct conditioning on the audio signal alongside textual prompts to perform ASR. While large audio language models (LALMs) are designed for general audio understanding with diverse prompts, LLM-based ASR systems focus specifically on transcription, relying on one fixed , manually defined prompt to be used during both training and inference [ 3 , 11 , 12 , 13 , 14 , 15 , 4 , 16 , 17 ] . This fixed-prompt setup ensures alignment between training objectives and inference behavior, making it well-suited for applications where high-accuracy transcription is the primary goal. While prior work has explored different speech encoders and projectors [ 2 , 18 , 19 ] , a critical component has been largely overlooked: the manually chosen prompt . To the best of our knowledge, no prior work has systematically studied its impact on ASR performance. In this paper, we take a first step toward understanding and improving prompt robustness in LLM-based ASR. We address two key research questions: (i) How important is the choice of the prompt for ASR performance? and (ii) Can we naturally extend a typical LLM-based ASR architecture to improve its robustness to prompt choice? To answer the first question, our comprehensive evaluation of prompts from recent literature across multiple datasets reveals that prompt choice can drastically impact ASR output, with certain prompts yielding significantly better performance even under identical model and data conditions. To address the second question, we propose a prompt projector as a simple, yet effective, architectural extension. Inspired by the success of the speech projector, we hypothesize that a similar projection function can align prompt embeddings to a more effective region of the LLM input space. Our approach is distinct from soft-prompt learning, as it learns a common projection rather than individual embeddings. Our main contributions are the following: (1) We provide the first systematic analysis of prompt sensitivity in LLM-based ASR systems; (2) we extend the original architecture by introducing the prompt projector; (3) we validate it across five evaluation sets, yielding consistently increased robustness and mitigating prompt-induced performance variance; (4) we release our source code for reproducibility. 1 1 1 https://github.com/idiap/llm-asr-prompt Figure 1 : Typical LLM-based ASR system composed of a fixed prompt, a speech encoder, and an LLM connected by a speech projector, sp(\cdot) . The proposed extension, highlighted in orange, introduces a learnable prompt projector, pp(\cdot) , into the original (frozen) architecture. The pp(\cdot) learns a common/single projection to transform all the original prompt embeddings ( \blacktriangle ) into more effective ones ( \blacktriangle ). 2 Methodology 2.1 Base Model As our base model, we adopt the recently proposed SLAM-ASR architecture for LLM-based ASR [ 3 ] .This model showed that a simple speech projection module is sufficient to achieve competitive performance, outperforming more complex LLM-based ASR systems. Its strong results and minimal design make it an ideal foundation for our experiments. Specifically, the speech projector, sp(\cdot) , is defined as: \mathbf{e}_{i}=sp(\mathbf{z}_{i})=\text{ReLU}(\mathbf{z}_{i}W_{1}+b_{1})W_{2}+b_{2}, (1) where \mathbf{z}_{i} denotes the i -th downsampled audio feature, obtained by concatenating k consecutive encoded frames along the temporal dimension. The resulting projection \mathbf{e}_{i} matches the dimensionality of the LLM input embeddings. In the original work [ 3 ] , an input audio consisting of n downsampled features, \mathbf{z}_{1},\dots,\mathbf{z}_{n} , is fed to the LLM using the following prompt, which we refer to as the “base” prompt: 2 2 2 The original paper places speech embeddings after "USER:" , but the released code prepends them. Our prompt here reflects their implementation. {speech} <s>USER: Transcribe speech to text. ASSISTANT: { transcript } </s> where { transcript } is the transcription corresponding to the audio, which is either provided during training or forced to be generated by the LLM at inference time. Here, {speech} is replaced by the sequence of n projected speech embeddings, sp(\mathbf{z}_{1}),\dots,sp(\mathbf{z}_{n}) , when the prompt is provided to the LLM. We use the same speech encoder and LLM as in the original work, as this configuration has been shown to outperform models using larger and more recent LLMs [ 3 , 16 , 12 ] , offering a good trade-off between ASR performance and computational efficiency: \bullet WavLM-large 3 3 3 https://huggingface.co/microsoft/wavlm-large : speech encoder trained on 94k hours of unlabeled data using self-supervision [ 20 , 21 , 22 , 23 ] . \bullet Vicuna-7B 4 4 4 https://huggingface.co/lmsys/vicuna-7b-v1.5 : LLM fine-tuned from LLama with SFT and optionally RLHF [ 24 ] . 5 5 5 We tested Llama 3 8B, but Vicuna 7B still outperformed it on 3 of the 5 evaluation sets, so we stick to the original paper’s setup. Throughout all experiments, we use the WavLM-large + Vicuna-7B combination. This configuration was shown to be optimal in WER without fine-tuning the speech encoder [ 3 ] . We refer to this model as the “vanilla” model and, as in the original work, it is trained by only learning the projector (Equation