The Impact of Model Scaling on Seen and Unseen Language Performance

Rhitabrat Pokharel, Sina Bagheri Nezhad, Ameeta Agrawal, Suresh Singh · 2025

Read The Impact of Model Scaling on Seen and Unseen Language Performance by Rhitabrat Pokharel, Sina Bagheri Nezhad, Ameeta Agrawal, Suresh Singh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05629.txt.

Introduction

Big language models speak many languages, but whether making them bigger helps every language equally is an open question. Rhitabrat Pokharel, Sina Bagheri Nezhad, Ameeta Agrawal, and Suresh Singh test three model families at several sizes across 204 languages. They probe text sorting and translation, both for languages the models saw in training and for ones they never did.

With no examples given, bigger models barely improved at all, and unseen languages lagged far behind seen ones. With just two examples, larger models improved steadily and clearly on the sorting task. Translation gained from size only in the instruction-tuned model. The authors also find that the total resources behind a language predict success better than its exact share of the training mix.

Catalog record

CREATORRhitabrat Pokharel, Sina Bagheri Nezhad, Ameeta Agrawal, Suresh Singh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-05629

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Impact of Model Scaling on Seen and Unseen Language Performance Rhitabrat Pokharel, Sina Bagheri Nezhad, Ameeta Agrawal, Suresh Singh Abstract The rapid advancement of Large Language Models (LLMs), particularly those trained on multilingual corpora, has intensified the need for a deeper understanding of their performance across a diverse range of languages and model sizes. Our research addresses this critical need by studying the performance and scaling behavior of multilingual LLMs in text classification and machine translation tasks across 204 languages. We systematically examine both seen and unseen languages across three model families of varying sizes in zero-shot and few-shot settings. Our findings show significant differences in scaling behavior between zero-shot and two-shot scenarios, with striking disparities in performance between seen and unseen languages. Model scale has little effect on zero-shot performance, which remains mostly flat. However, in two-shot settings, larger models show clear linear improvements in multilingual text classification. For translation tasks, however, only the instruction-tuned model showed clear benefits from scaling. Our analysis also suggests that overall resource levels, not just the proportions of pretraining languages, are better predictors of model performance, shedding light on what drives multilingual LLM effectiveness. Portland State University Oregon, USA {pokharel, sina5, ameeta, singhsp}@pdx.edu Introduction Current trends in the development of large language models (LLMs) emphasize the use of extensive data in several different languages and larger model sizes ( Srivastava et al. 2023 ) . Scaling trends serve to quantify the connection between a model’s performance and critical design elements such as training data size, number of model parameters, or architectural intricacies, providing invaluable insights for model refinement, resource distribution, and the selection of pertinent training data. Although significant research efforts have focused on scaling trends within predominantly English settings ( Radford et al. 2019 ; Lin et al. 2022 ; McKenzie et al. 2023 ; Wei et al. 2023 ; Scao et al. 2023 ; Chia et al. 2024 ) , their understanding in multilingual contexts remains underexplored ( Sun and Miceli-Barone 2024 ) . Several massively multilingual LLMs have been introduced ( Lin et al. 2022 ; Achiam et al. 2023 ; Scao et al. 2023 ; Shliazhko et al. 2024 ) , however, evaluation is often limited to a few tens of languages, limiting the large-scale evaluation of current multilingual language models in many languages, especially those truly low-resource languages ( Ahuja et al. 2023 ) . Moreover, prior work focuses on studying scaling on languages that were seen during pretraining whereas we extend our focus also to languages that were potentially not seen during pretraining. In this paper, we take a step towards performing a large-scale study of the scaling behavior of multilingual models from the lens of multilingual text classification as well as text generation. We look into the relation between model sizes (number of parameters) and the downstream task performance across diverse languages and resource levels. We comprehensively evaluate several different model sizes from three model families across 204 low and high resource languages in two types of tasks – topic classification and machine translation. Multilingual topic classification was chosen for two key reasons: it is a widely studied and popular task in natural language processing (NLP) ( Scao et al. 2023 ) , and the new parallel dataset SIB-200 ( Adelani et al. 2024 ) enables consistent and unified analysis across 204 languages. Similarly, machine translation with the FLORES-200 dataset allows for comparable evaluation under text generation setting. Specifically, we assess the performance of XGLM (with model sizes ranging from 564 million to 7.5 billion parameters), bloom , and bloomz models (ranging from 560 million to 7.1 billion parameters). In total, the results of our study are derived from evaluation involving more than 2 million instances. To our knowledge, ours is the first study to investigate the impact of multilingual scaling across more than 200 languages, both seen as well as unseen 1 1 1 A language is considered as potentially unseen if it is not listed in the language distribution of the model’s pretraining data. , across two different tasks. We seek to answer the following three questions: 1) What is the impact of model scaling on seen and unseen languages of different resource levels for each type of task? 2) How do different settings (zero-shot and few-shot in-context learning) affect the overall model effectiveness? 3) What is the correlation between general resource level, the language-specific training data and performance? Model Model Sizes (# parameters) # Pretrain Tokens # Lang. / # Lang. Families Vocab. xglm 564M, 1.7B, 2.9B, 7.5B 500B 30 / 15 256,008 bloom 560M, 1.1B, 1.7B, 3B, 7.1B 341B 46 / 8 250,880 bloomz 560M, 1.1B, 1.7B, 3B, 7.1B 341B 46 / 8 250,880 Table 1: Overview of model details. Our key observations are as follows: • In multilingual settings, scaling laws do not apply consistently. Scaling depends on the task and the inference setting – for example, we see scaling in 2-shot classification tasks but not in zero-shot classification or in zero-shot and 2-shot machine translation. • While larger models consistently leverage the benefits of few-shot ICL for classification tasks for both seen as well as unseen languages, smaller models struggle in few-shot ICL settings instead yielding better results in zero-shot settings, especially for unseen languages and low resource languages that were seen during the pretraining. • For the generative task, ICL appears to hurt two language models ( xglm and bloomz ) while only slightly benefiting the third ( bloom ), regardless of model scale. • For both tasks, we find that performance obtains stronger correlations with general resource levels rather than language-specific training data, which subsequently result in more pronounced disparities in model performance across different resource levels. Related Work The impact of model scale on language model performance, particularly within English settings, has been extensively explored in recent years with several studies reinforcing the finding that larger models generally result in enhanced performance ( Radford et al. 2019 ; Hestness et al. 2017 ; Kaplan et al. 2020 ; Rae et al. 2022 ; Wei et al. 2022 ; Hoffmann et al. 2022 ; Smith et al. 2022 ; Chowdhery et al. 2022 ) . Despite the relationship between model scale and performance not always being linear or predictable ( Wei et al. 2022 ; Wei et al. 2023 ; Xia et al. 2023 ) , larger models continue to be developed, including several multilingual large language models such as GPT-3 ( Brown et al. 2020b ) , XGLM ( Lin et al. 2022 ) , BLOOM ( Scao et al. 2023 ) , LLaMA ( Touvron et al. 2023 ) , PaLM 2 ( Anil et al. 2023 ) , and others. Although massive multilingual models benefit from positive transfer across languages, the performance of a model deteriorates as its language coverage expands, a challenging phenomenon also known as the “curse of multilinguality” ( Conneau et al. 2020 ; Pfeiffer et al. 2022 ) . Despite the scaling of multilingual models, research on the impact of scaling on performance remains limited. Lin et al. (2022) showed that different tasks lead to different scaling behavior. However, the datasets in their study were limited to at most 15

Read the complete text