Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer

Seungyoon Lee, Seongtae Hong, Hyeonseok Moon, Heuiseok Lim · 2025

Read Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer by Seungyoon Lee, Seongtae Hong, Hyeonseok Moon, Heuiseok Lim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10945.txt.

Introduction

Seungyoon Lee and co-authors present a method from 2025 for moving large language models across languages. Most transfer methods swap in a new word list but lose some skill in the new language. The team calls their method Semantic Aware Linear Transfer, or SALT for short. It reuses word data from models already trained in the target language.

SALT builds a fit line for each word that the two word lists do not share. The fit is guided by how close words are in meaning. Tests across tasks show gains over older swap methods. The work offers a cheaper path to strong models in more languages.

Catalog record

CREATORSeungyoon Lee, Seongtae Hong, Hyeonseok Moon, Heuiseok Lim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10945

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer Seungyoon Lee, Seongtae Hong, Hyeonseok Moon, Heuiseok Lim Abstract Large Language Models (LLMs) increasingly incorporate multilingual capabilities, fueling the demand to transfer them into target language-specific models. However, most approaches, which blend the source model’s embedding by replacing the source vocabulary with the target language-specific vocabulary, may constrain expressive capacity in the target language since the source model is predominantly trained on English data. In this paper, we propose S emantic A ware L inear T ransfer ( SALT ), a novel cross-lingual transfer technique that recycles embeddings from target language Pre-trained Language Models (PLMs) to transmit the deep representational strengths of PLM-derived embedding to LLMs. SALT derives unique regression lines based on the similarity in the overlap of the source and target vocabularies, to handle each non-overlapping token’s embedding space. Our extensive experiments show that SALT significantly outperforms other transfer methods and achieves lower loss with accelerating faster convergence during language adaptation. Notably, SALT obtains remarkable performance in cross-lingual understanding setups compared to other methods. Furthermore, we highlight the scalable use of PLMs to enhance the functionality of contemporary LLMs by conducting experiments with varying architectures. 1 Introduction As Large Language Models (LLMs) continue to demonstrate remarkable performance across various knowledge-based benchmarks and sub-tasks, the demand for robust linguistic capabilities in multilingual or language-specific contexts has surged Wei et al. (2022a) ; Wei et al. (2022b) ; Peng et al. (2023) ; Taori et al. (2023) ; Zhou et al. (2024) . However, most multilingual-considered LLMs show limited practicality in target languages due to their English-centric training and reliance on common vocabulary designed to incorporate a wide range of languages Puttaparthi et al. (2023) ; Le Scao et al. (2023) ; Lai et al. (2023a) ; Zhao et al. (2024) ; Team et al. (2024) ; Dubey et al. (2024) . This drawback also correlates with the large portion of their parameters, which are the embeddings of irrelevant language tokens to target language. To address this challenge, a promising line of studies has focused on cross-lingual transfer Artetxe et al. (2020) ; Tran (2020) ; Gee et al. (2022) ; Dobler and De Melo (2023) ; Remy et al. (2024) ; Liu et al. (2024) ; Mundra et al. (2024) ; Ye et al. (2024) . These efforts include replacing a source vocabulary with a target language vocabulary, simply initializing embeddings with newly manipulated embeddings stem from source model Minixhofer et al. (2022) ; Dobler and De Melo (2023) ; Liu et al. (2024) ; Mundra et al. (2024) . However, a newly initialized embedding that relies solely on the source model inevitably experiences limited expressiveness in the target language due to the immature learning process of the source model, which does not mainly concentrate on the target language. Moreover, most approaches adopt small encoder architectures such as BERT-like models Devlin et al. (2019) ; Liu (2019) ; Conneau et al. (2020) as a source model, leaving their applicability on contemporary decoder-based LLMs unexplored Gogoulou et al. (2022) ; Gee et al. (2022) ; Zeng et al. (2023) ; Dobler and De Melo (2023) ; Ye et al. (2024) . In this work, we propose S emantic A ware L inear T ransfer ( SALT ), a novel cross-lingual transfer method that leverages the rich representational power of target language embeddings in traditional Pre-trained Language Models (PLMs) 2 2 2 In this work, we define PLMs to small-scale language models with only millions of parameters, developed prior to the advent of LLMs. to convert English-centric LLMs into target language–specialized LLMs. To facilitate the transfer, we identify the semantically closest tokens for each non-shared token from a shared vocabulary space. Based on these semantically similar tokens, we fit a unique linear regression to transfer the embeddings from the PLM’s space to the LLM’s space. This procedure enables non-shared vocabulary to be transferred to LLM space while retaining the semantic representation embedded in PLM embedding. In our experiments, we focus on investigating the potential of transferred embeddings from PLMs in three aspects: (i) whether the transferred embeddings can be well-aligned with inherent knowledge in the source model, (ii) the influence on better initialization to the target language and convergence during continual pre-training, and (iii) the ability of understanding between target language and mainstream language in cross-lingual environments. We empirically demonstrate that embeddings from target language PLMs can be more helpful for cross-lingual transfer in LLMs than existing methods. SALT not only surpasses various strong baselines in downstream tasks but also provides a better initialization for target language adaptation with faster convergence during continual pre-training. Notably, we find that SALT preserves English capability more effectively while maintaining alignment between English and the target language in a cross-lingual setup. Furthermore, by conducting additional study on various PLM architectures (Encoder, Decoder, and Encoder-Decoder), we discover that SALT can be extended to PLMs with various architectures and can even be a valid strategy for basic models such as BERT Devlin et al. (2019) . Our contributions are as follows: • We propose SALT, a novel embedding transfer method to effectively project the deep representation capabilities of PLM embeddings onto LLMs based on semantic information. • We empirically verify that SALT has superiority in cross-lingual environments as well as downstream tasks and language modeling for the target language. • We further show the versatility of SALT in experiments with various types of PLMs. • By recycling previously prominent PLMs as target language embeddings, we demonstrate the potential scalability and advantage of PLMs, and propose a new approach to leverage PLMs in the era of LLMs. 2 Related Work Cross-lingual transfer to a target language includes approaches that manipulate the vocabulary or initialize embedding for the new target vocabulary. In vocabulary manipulation, most research has considered adding target language relevant vocabulary Wang et al. (2020) ; Chau et al. (2020) ; Cui et al. (2023) ; Larcher et al. (2023) ; Fujii et al. (2024) ; Mundra et al. (2024) ; Yamaguchi et al. (2024b) ; Zhao et al. (2024) . Vocabulary expansion is widely adopted for developing language-specific models from a source model Balachandran (2023) ; Cui et al. (2023) ; Fujii et al. (2024) . This line of work requires access to a large amount of target language data. In light of this, Yamaguchi et al. (2024a) delve into cross-lingual vocabulary expansion in low-resource settings across initialization approaches and training strategies. Also, Zhao et al. (2024) investigate training scales required for vocab extension and the influence of transfer on capabilities of language generation and following instructions. Figure 1: Summary of SALT. By using the paired embeddings of semantically similar tokens for each non-shared token, we create a unique least square matrix X_{t_{i}} to transfer from PLM to LLM. k denotes the number of selected nearest tokens among overlapping tokens for linear transfer, and h_{t} and h_{s} refer to the hidden dimensions of the target and source models. On the other hand, given that overall cost increase derived from extended vocabulary, several cross-lingual transfer studies have pursued replacing the original vocabulary with a new target vocabulary Zeng et al. (2023) ; Ostendorff and Rehm

Read the complete text