Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation

Luca Moroni, Giovanni Puccetti *, Pere-Lluis Huguet Cabot, Andrei Stefan Bejgu, Edoardo Barba, Alessio Miaschi, Felice Dell’Orletta, Andrea Esuli, Roberto Navigli · 2025

Read Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation by Luca Moroni, Giovanni Puccetti *, Pere-Lluis Huguet Cabot, Andrei Stefan Bejgu, Edoardo Barba, Alessio Miaschi, Felice Dell’Orletta, Andrea Esuli, Roberto Navigli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17025.txt.

Introduction

Large language models work best in English, and a 2025 paper by Luca Moroni and co-authors asks how to tune them for Italian. Models built for English split Italian words into too many small parts, which slows them down. The team compares many ways to swap in Italian word parts.

Their new method uses a neural map to link new word parts to old ones with like meanings. Tests on two well-known models cut the split rate and cut one model by a large share of its size. Short extra training then brought back most task skill. Readers learn a low-cost path to strong models for one more language.

Catalog record

CREATORLuca Moroni, Giovanni Puccetti *, Pere-Lluis Huguet Cabot, Andrei Stefan Bejgu, Edoardo Barba, Alessio Miaschi, Felice Dell’Orletta, Andrea Esuli, Roberto Navigli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-17025

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation Luca Moroni, Giovanni Puccetti *, Pere-Lluis Huguet Cabot, Andrei Stefan Bejgu, Edoardo Barba, Alessio Miaschi, Felice Dell’Orletta, Andrea Esuli, Roberto Navigli Abstract The number of pretrained Large Language Models (LLMs) is increasing steadily, though the majority are designed predominantly for the English language. While state-of-the-art LLMs can handle other languages, due to language contamination or some degree of multilingual pretraining data, they are not optimized for non-English languages, leading to inefficient encoding (high token "fertility") and slower inference speed. In this work, we thoroughly compare a variety of vocabulary adaptation techniques for optimizing English LLMs for the Italian language, and put forward Semantic Alignment Vocabulary Adaptation (SAVA), a novel method that leverages neural mapping for vocabulary substitution. SAVA achieves competitive performance across multiple downstream tasks, enhancing grounded alignment strategies. We adapt two LLMs: Mistral-7B-v0.1 , reducing token fertility by 25%, and Llama-3.1-8B , optimizing the vocabulary and reducing the number of parameters by 1 billion. We show that, following the adaptation of the vocabulary, these models can recover their performance with a relatively limited stage of continual training on the target language. Finally, we test the capabilities of the adapted models on various multi-choice and generative tasks. 1 1 1 We release our code and models at https://github.com/SapienzaNLP/sava 1 Introduction Figure 1: Fertility for two different tokenizers, Mistral-7B-v0.1 (left) and Minerva (right), over Italian texts from CulturaX (blue) and Wikipedia (red). Large Language Models (LLMs) have gained immense popularity and are increasingly being utilized across a wide range of applications ( Radford et al., 2019 ; Kojima et al., 2022 ) . Despite their impressive performance, these models are mainly English-centric, that is, most state-of-the-art models are designed and pre-trained on datasets with a primary focus on English ( Jiang et al., 2023 ; Dubey et al., 2024 ; Mesnard et al., 2024 ) . Although native multilingual models – i.e. fully pre-trained in multiple target languages – have been released over the years ( Le Scao et al., 2023 ) , they still fall short of achieving performance levels comparable to models pre-trained in English. The primary challenge is addressing underrepresented languages, where large, clean, open-access text corpora are often scarce ( Weber et al., 2024 ; Nguyen et al., 2024 ) . This scarcity is problematic because models require vast amounts of high-quality data to achieve satisfactory performance ( Hoffmann et al., 2022 ) . Moreover, multilingual models generally reach sub-optimal performance due to the well-known curse of multilinguality ( Conneau et al., 2020 ) . A promising solution to these challenges is the adaptation of pretrained English LLMs to other languages ( Chau et al., 2020 ) . Recent studies highlight that fine-tuning English-centric models to support other languages yields substantial benefits, allowing for efficient adaptation while minimizing computational resources and training time. This method reduces both the training budget and the number of tokens required, demonstrating competitive performance even in low-resource scenarios ( Koto et al., 2021 ; Minixhofer et al., 2022 ; Gee et al., 2022 ; Ostendorff and Rehm, 2023 ) . Another important aspect, alongside the downstream performance of language models, is the tokenizer’s fertility in target languages. LLMs rely on a tokenizer, which is trained on a mix of text (either LLM’s training data or not), transforming raw text into word-piece tokens; fertility is the average number of tokens in which a word is split ( Brown et al., 1993 ) . The fertility of a tokenizer is highly sensitive to the language and the type of text it was trained on, as well as the text on which the fertility itself is measured. Figure 1 shows an example of this phenomenon comparing the fertility of Minerva-LLMs , a family of Italian-first LLMs ( Orlando et al., 2024 ) , and Mistral-7B-v0.1 , an English-first LLM ( Jiang et al., 2023 ) , on two corpora in Italian. In this work, we explore the adaptation of two state-of-the-art English LLMs to the Italian language using both vocabulary adaptation and continual learning. Additionally, we introduce a novel vocabulary adaptation technique called Semantic Alignment Vocabulary Adaptation (SAVA) and conduct a comprehensive comparison with recent approaches ( Gee et al., 2022 ; Ostendorff and Rehm, 2023 ) , examining the impact of vocabulary substitution on model performance throughout the adaptation process. After the adaptation of the vocabulary, when tokenizing Italian texts, we are able to reduce the fertility of Mistral-7B-v0.1 by 25% and of Llama-3.1-8B by 16%. As regards Mistral-7B-v0.1 , we do not increase its vocabulary size or model parameters, while for Llama-3.1-8B , we effectively reduce its vocabulary size by 75% thereby reducing the final model size by 10%. Overall, we reduce memory and compute footprint of the models. To summarize, the main contributions are: • Introducing an effective approach for adapting tokenizers and vocabularies of generative models, leading to competitive performance over existing methods across several downstream benchmarks; • Providing a detailed comparative analysis of various tokenizer adaptation techniques, with a focus on continual training in low- to mid-resource scenarios. • Analyzing the embedding representations learned through different adaptation techniques, offering a deeper understanding of how vocabulary modifications impact model performance and generalization. 2 Related Work Language-Adaptive Pretraining Designing LLMs in a target language and thus training them from scratch is the best approach to obtain an adequate token fertility from the outset and minimize interference from pretrained data on different languages. However, this approach is often impractical, especially in low-resource settings and on a low computational budget. For this reason, several recent studies ( de Vries and Nissim, 2021 ; Gee et al., 2022 ; Csaki et al., 2024 ) have focused on the adaptation of pretrained LLMs to new languages. Pretrained LLMs can be adapted to a specific language using a small quantity data compared to what is needed in the pretraining stage. A straightforward approach to achieving this is Language-Adaptive Pre-Training (LAPT), utilized by Chau et al. (2020) in a multilingual setting where they tested continual training of multilingual LLMs on target languages. Interestingly, LAPT was previously proposed on encoder-only architectures by Gururangan et al. (2020) , where they successfully adapted RoBERTa ( Zhuang et al., 2021 ) models in a biomedical domain. In LAPT, models do not undergo any structural change to their architecture. This usually results in performance improvements, however it does not address the limitations of using a sub-optimal tokenizer that is less suited to the encoding of different languages. Regarding LAPT research in English-to-Italian models, there have been several attempts, most notably LLaMAntino-2-LLMs , which is a fine-tuning of LLama 2 on Italian translated conversations ( Basile et al., 2023 ) , and LLaMAntino-3-ANITA-8B-Inst-DPO-ITA , a more recent effort that is built upon Llama-3-8B using a similar approach ( Polignano et al., 2024 ) . Vocabulary Adaptation Techniques To tackle the fertility issue, recent research has focused on improving language adaptation by modifying the tokenizer and vocabulary of pretrained LLMs to better fit the target language. Several efforts in this area have shown the effectiveness of vocabulary adaptation techniques. Minixhofer et al. (2022) and Liu et al. (2024) propose to replace the

Read the complete text