Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation
Snegha A, Sayambhu Sen, Piyush Singh Pasi, Abhishek Singhania, Preethi Jyothi · 2025
Read Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation by Snegha A, Sayambhu Sen, Piyush Singh Pasi, Abhishek Singhania, Preethi Jyothi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24619.txt.
Introduction
Snegha A, Sayambhu Sen, Piyush Singh Pasi, Abhishek Singhania, and Preethi Jyothi test ways to teach a model a new task in English only. The goal is for the skill to carry to more than thirty five other tongues with no more training. They focus on soft prompt styles, which are less used than the more known low rank method.
With two large models, the prompt styles beat the low rank base lines by up to six points on one reading test. Gains held across tongue groups, scripts, and model sizes. One style used just over one million learned weights, yet still gave steady gains.
Catalog record
| CREATOR | Snegha A, Sayambhu Sen, Piyush Singh Pasi, Abhishek Singhania, Preethi Jyothi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24619
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation Snegha A, Sayambhu Sen, Piyush Singh Pasi, Abhishek Singhania, Preethi Jyothi Abstract With the release of new large language models (LLMs) like Llama and Mistral, zero-shot cross-lingual transfer has become increasingly feasible due to their multilingual pretraining and strong generalization capabilities. However, adapting these decoder-only LLMs to new tasks across languages remains challenging. While parameter-efficient fine-tuning (PeFT) techniques like Low-Rank Adaptation (LoRA) are widely used, prefix-based techniques such as soft prompt tuning, prefix tuning, and Llama Adapter are less explored, especially for zero-shot transfer in decoder-only models. We present a comprehensive study of three prefix-based methods for zero-shot cross-lingual transfer from English to 35+ high- and low-resource languages. Our analysis further explores transfer across linguistic families and scripts, as well as the impact of scaling model sizes from 1B to 24B. With Llama 3.1 8B, prefix methods outperform LoRA-baselines by up to 6% on the Belebele benchmark. Similar improvements were observed with Mistral v0.3 7B as well. Despite using only 1.23M learning parameters with prefix tuning, we achieve consistent improvements across diverse benchmarks. These findings highlight the potential of prefix-based techniques as an effective and scalable alternative to LoRA, particularly in low-resource multilingual settings. 1 Introduction Large language models (LLMs) exhibit strong multilingual and zero-shot generalization abilities due to exposure to diverse pretraining data. Nonetheless, cross-lingual transfer remains challenging given the linguistic diversity and complexity of adapting large models efficiently without significant computational overhead. To address the high computational and memory costs of full model finetuning, recent advances in parameter-efficient finetuning (PeFT) techniques focus on updating only a small subset of model parameters while keeping the majority of the pretrained weights frozen. This design significantly reduces the adaptation cost and makes large-scale models more practical for multilingual and domain-specific applications. Methods such as Low-Rank Adaptation (LoRA) Hu et al. 2022 and instruction-tuned adapters have shown promising results in efficiently tailoring models to new tasks without requiring extensive resources. Among the various PeFT techniques, prefix-based approaches like soft prompting Lester et al. 2021 ; Liu et al. 2024a and prefix tuning Li and Liang 2021 are particularly compelling because they introduce learnable components either at the input or within the transformer stack, enabling flexible task adaptation without altering the underlying architecture of the model. While these prefix-based techniques have been shown to be effective in monolingual scenarios and task-specific settings, their potential in facilitating zero-shot cross-lingual transfer is under-explored. This is especially relevant for decoder-only LLMs, which are increasingly being deployed in multilingual environments. Unlike encoder-decoder models that have been more thoroughly studied for transfer across languages, decoder-only models present unique challenges due to their reliance on autoregressive decoding. Understanding whether prefix-based PeFT methods can enhance zero-shot cross-lingual performance in such models has not been previously studied in detail. In this work, we provide the first systematic study of prefix-based PeFT methods for zero-shot cross-lingual transfer in decoder-only LLMs . Our contributions can be summarized as follows: • We evaluate prefix-based adaptation on models ranging from 1B parameters to large-scale 24B models to show the effectiveness of prefix tuning in multilingual transfer across models of varying sizes. • Our study spans four well-recognized multilingual benchmarks – XQUAD, XNLI, Belebele and MGSM – to compare the performance of LoRA and prefix-based tuning. • We provide a detailed comparison of prefix-based methods (soft prompts, prefix tuning, LLaMA-Adapter) against LoRA and full fine-tuning 1 1 1 Due to computational limitations, full fine-tuning is restricted to the SQuAD dataset on Llama 3.1 8B. , systematically analyzing their strengths and limitations across tasks and 35+ high- and low-resource languages. Additionally, we investigate transfer patterns across linguistic families and scripts. Together, our findings position prefix-based adaptation as a lightweight yet powerful strategy for cross-lingual and reasoning-oriented applications, particularly in resource-constrained multilingual settings. Figure 1: Schematic representation of: (A) LoRA fine-tuning and prefix-based methods, (B) Llama Adapter, (C) Prefix tuning, and (D) Soft prompt tuning. 2 Related Work Cross-lingual transfer is a key challenge in multilingual NLP. It is traditionally tackled through full fine-tuning of multilingual models. However, with large decoder-only LLMs like Llama and Mistral, full fine-tuning is costly, leading to PeFT approaches. LoRA Hu et al. 2022 introduces low-rank trainable matrices into frozen weights to reduce training overhead. Alternatively, prefix-based methods either add learnable tokens at the input layer Lester et al. 2021 ; Liu et al. 2024a or to attention keys and values at each layer Li and Liang 2021 , enabling efficient task adaptation. Soft prompt tuning has been extensively studied for cross-lingual transfer in encoder and encoder-decoder models, particularly in classification tasks. For instance, Philippy et al. 2024 demonstrated that soft prompts can generalize better across languages with fewer parameters, following the “less is more" principle. Similarly, Philippy et al. 2025 utilized multilingual verbalizers and contrastive label smoothing to further enhance cross-lingual classification. Recent work such as Vykopal et al. 2025 introduced language-specific soft prompts specifically designed for transfer learning, showing that combining language-specific and task-specific prompts improves generalization. However, these prior works predominantly used multilingual encoder-only and encoder-decoder models, and appended prefix tokens only to the input. As soft prompts have several limitations in effectively adapting models to new tasks, prefix tuning emerged as a promising approach. Cross-lingual alignment through prompt-based pretraining, as proposed by Tu et al. 2024 , further improved intent classification and slot-filling performance but it is not a zero-shot setting (as in our work). A recent variant of prefix tuning is LLaMA Adapter Dubey et al. 2024 that introduced zero-initialized attention mechanisms for efficient prefix training and achieved strong instruction-following capabilities; however, they did not evaluate on any multilingual benchmarks. A related line of work has focused on extending prefix tuning to instance-specific adaptation based on the input prompt for improved model performance Liu et al. 2024c ; Jiang et al. 2022 ; Liu et al. 2024b ; Zhu et al. 2024 . Few comparative studies have examined parameter-efficient tuning for multilingual settings, and most have been restricted to encoder-only models or small decoder-only models with only a few million parameters. For instance, Zhao and Schütze 2021 systematically compared discrete prompting, soft prompting, and fine-tuning on the few-shot multilingual NLI task using XLM-RoBERTa-base. Similarly, Tu et al. 2022 compared prompt tuning with fine-tuning across diverse NLU tasks on XLM-R and mBERT. Tu et al. 2022 evaluate prefix tuning on the encoder-only XLM-R model and showed its effectiveness over full fine-tuning in zero-shot cross-lingual transfer. Tu et al. 2022 investigated a decoder-based multilingual model (XGLM), but their analysis was limited to a single small model. They showed that prompt tuning can sometimes surpass fine-tuning, particularly for low-resource languages, although performance remained highly sensitive to the underlying tokenization scheme. Our work significantly extends their analysis to large decoder-only LLMs and presents a comprehensive comparison of multiple prefix-based methods, including soft prompts, prefix tuning, and LLaMA Adapter. 3 Methodology Low-Rank Adaptation (LoRA). LoRA Hu et al. 2022 is a parametric fine-tuning technique that has become one of the most popular approaches to enable cross-lingual transfer in LLMs. It introduces trainable low-rank matrices, typically in the query, key, and value projections, while keeping the base model frozen. These learned matrices are added to the original weights