Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models
Ahmed Elshabrawy Thanh-Nhi Nguyen Yeeun Kang Lihan Feng, Annant Jain Faadil A. Shaikh Jonibek Mansurov Mohamed Fazli Imam, Jesus-German Ortiz-Barajas Rendi Chevi Alham Fikri Aji · 2025
Read Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models by Ahmed Elshabrawy Thanh-Nhi Nguyen Yeeun Kang Lihan Feng, Annant Jain Faadil A. Shaikh Jonibek Mansurov Mohamed Fazli Imam, Jesus-German Ortiz-Barajas Rendi Chevi Alham Fikri Aji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01592.txt.
Introduction
Elshabrawy and co-authors test whether small encoder models can jump between languages the way giant chat models do. Encoders like BERT are cheap to run but historically weak at zero-shot tasks. Statement tuning fixes that by recasting each task as statements to judge, and their 2025 study extends the trick across languages.
State-of-the-art encoders, tuned this way, generalized across languages well, rivaling multilingual giants while using far less memory and compute. The authors study how the tuning data set should be built, where the efficiency gains land, and which languages transfer best. For low-resource languages, where big models are costly or absent, that trade matters. Code and models are released. The paper makes the case that inclusive language technology may not need the biggest models after all.
Catalog record
| CREATOR | Ahmed Elshabrawy Thanh-Nhi Nguyen Yeeun Kang Lihan Feng, Annant Jain Faadil A. Shaikh Jonibek Mansurov Mohamed Fazli Imam, Jesus-German Ortiz-Barajas Rendi Chevi Alham Fikri Aji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01592
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models Ahmed Elshabrawy Thanh-Nhi Nguyen Yeeun Kang Lihan Feng, Annant Jain Faadil A. Shaikh Jonibek Mansurov Mohamed Fazli Imam, Jesus-German Ortiz-Barajas Rendi Chevi Alham Fikri Aji Abstract Large Language Models (LLMs) excel in zero-shot and few-shot tasks, but achieving similar performance with encoder-only models like BERT and RoBERTa has been challenging due to their architecture. However, encoders offer advantages such as lower computational and memory costs. Recent work adapts them for zero-shot generalization using Statement Tuning, which reformulates tasks into finite templates. We extend this approach to multilingual NLP, exploring whether encoders can achieve zero-shot cross-lingual generalization and serve as efficient alternatives to memory-intensive LLMs for low-resource languages. Our results show that state-of-the-art encoder models generalize well across languages, rivaling multilingual LLMs while being more efficient. We also analyze multilingual Statement Tuning dataset design, efficiency gains, and language-specific generalization, contributing to more inclusive and resource-efficient NLP models. We release our code and models 1 1 1 https://github.com/mbzuai-nlp/Multilingual-ST . 1 Introduction Large Language Models (LLMs) have shown great capabilities in zero-shot and few-shot settings Radford et al. 2019 ; Brown et al. 2020 ; Artetxe et al. 2022 . However, these capabilities are often more difficult to observe in encoder-only models like BERT Devlin et al. 2019 and RoBERTa ( Liu et al. 2019 ) due to their architectural design. These models are typically pretrained with a Masked Language Modeling (MLM) objective and are fine-tuned by adding task-specific layers to enable their usage on a downstream task. These task-specific layers block the extension of these models to new tasks in a few-shot or zero-shot manner. Despite these difficulties, applying encoder models for zero-shot task generalization offers several advantages. First, encoder models are typically more lightweight than LLMs and therefore require less computational power and memory. Further, encoder models excel at generating contextual embeddings that better capture semantic meaning. For instance, recent work ( Qorib et al. 2024 ) demonstrated that decoder-only LLMs perform worse on word meaning comprehension than encoder-only models. Encoder-only models are also more efficient in inference for tasks such as sequence labeling due to their architecture ( Soltan et al. 2023 ) . Figure 1: Encoder models trained with Multilingual Statement-Tuning can generalize across new task and unseen languages during finetuning. To enable encoder model usage in zero-shot task generalization, Elshabrawy et al. 2025 introduced Statement-Tuning. This technique converts tasks into a set of statements with finite templates, training on an encoder-only model, RoBERTa, to discriminate between potential statements and derive final results. This method demonstrates the feasibility of using encoder models, typically specialized for specific tasks, to handle various unseen Natural Language Understanding (NLU) tasks, similar to zero-shot prompting in decoder models. It shows competitive performance compared to large language models (LLMs) with significantly fewer parameters and training data, highlighting its potential for zero-shot learning. However, the original approach focused only on English, raising questions about its applicability in multilingual settings and its ability to generalize to new tasks and languages. In this work, we aim to explore the adaptation of Statement-Tuning for multilingual NLP tasks. Specifically, we investigate whether encoder models can achieve zero-shot cross-lingual generalization similar to decoder-only LLMs Muennighoff et al. 2023 . Given the multilingual setting, it is crucial to emphasize the importance of low-compute solutions. Speakers and users of low-resource languages often lack the computational resources necessary to utilize memory-intensive LLMs. Therefore, our method, which leverages efficient encoder models, is particularly important as it offers a more accessible and inclusive approach to zero-shot text classification in these contexts Ruder 2022 . Hence our contributions are as follows: • We enable zero-shot generalization to unseen tasks and languages for encoder-only models, a capability typically limited to decoder-based models. • Our benchmarking shows that state-of-the-art multilingual encoder-only models match LLMs in performance while being more efficient. • We analyze multilingual Statement-Tuning dataset design, including language diversity and translated prompt templates. • We investigate when and how multilingual Statement-Tuning generalizes effectively across languages. • We compare Statement-Tuning inference speed and memory efficiency against generative models, showing significant advantages. 2 Related Work Zero-shot and Few-shot Approaches Using Encoder-Only Models There have been several works exploring prompt-based approaches to enable few-shot and zero-shot generalization in Encoder-only Models. Finetuning on cloze templates or label discrimination Schick and Schütze 2021 ; Gao et al. 2021 effectively utilizes encoder models for few-shot learning. Cloze templates have also been shown to fare better than regular finetuning in cross-lingual few-shot transfer to unseen languages from higher resourced languages Zhao and Schütze 2021 ; Tu et al. 2022 ; Ma et al. 2023 ; Ma et al. 2024 . However, to enable zero-shot task learning a reformulation of text classification tasks is necessary. Yin et al. 2019 introduce the reformulation of any zero-shot text classification in the form of entailment where statements can be formed from a series of choices and the correct choice is seen to be an entailment. Xu et al. 2023 use DeBERTa to show that the entailment formulation of zero-shot classification can be observed to be more effective than the generative approach employed by LLMs. Finally, Elshabrawy et al. 2025 propose Statement-Tuning to show that through template-based data augmentation, much smaller RoBERTa models can be finetuned on limited data to match or even exceed the zero-shot NLU capabilities of several LLMs of up to 70B parameters on monolingual classification tasks. While Elshabrawy et al. 2025 focused only on English, we studied whether Statement-Tuning method is possible in other languages. Additionally, we explore the efficiency of the approach in more detail and offer insight on the effect of pretraining data on the performance of Statement-Tuning. Zero-shot Prompting and Multitask Tuning While LLMs were shown to perform well on few-shot generalization Brown et al. 2020 , they showed less successful performance on zero-shot generalization. To tackle this issue, instruction tuning was proposed. Instruction tuning refers to fine-tuning language models on a collection of datasets described via instructions Wei et al. 2022 . Their model, Finetuned Language Net (FLAN), a decoder-only model of 137B parameters fine-tuned on more than 60 NLP datasets expressed via natural language instructions, proved effective in improving the zero-shot performance of models. They also showed that increasing the number of tasks involved in instruction tuning improves unseen task generalization performance and asserted that the benefits of instruction tuning are emerging abilities of language models (i.e., they emerge with sufficient scale). Subsequent work by Sanh et al. 2022 explored instruction tuning with T5 encoder-decoder models and proposed the T0 models and datasets. They fine-tuned T5 models of 3B and 11B parameters, which were smaller than the FLAN model but still within the billions-of-parameters range. Their findings established that with a more diverse prompt setup and an encoder-decoder model like T5, language models could achieve good performance with instruction tuning. Chung et al. 2022 found that instruction tuning is effective across a variety of model classes, such as PaLM, T5, and U-PaLM, as well as different prompting setups including zero-shot, few-shot, and chain-of-thought. Their models, FLAN-T5, ranged from 80M to 11B parameters and showed better performance than prior T5 checkpoints. Meanwhile, Mishra et al. 2022 ; Wang et al. 2022 ; Honovich et