Enhancing Multilingual Language Models for Code-Switched Input Data
Katherine Xie, Nitya Babbar, Vicky Chen, Yoanna Turura · 2025
Read Enhancing Multilingual Language Models for Code-Switched Input Data by Katherine Xie, Nitya Babbar, Vicky Chen, Yoanna Turura (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07990.txt.
Introduction
Katherine Xie, Nitya Babbar, Vicky Chen, and Yoanna Turura test language models on code-switched text in 2025. Many speakers blend two languages inside one chat, which strains models trained on single languages. The team asks whether pre-training on blended text helps.
They pre-trained a multilingual model on Spanish-English tweets from social media. The tuned model matched or beat the base model on four tasks, from grammar tagging to spotting names. Gains were largest for grammar tagging. A look inside the model showed its two language spaces pulled closer together, a clue for future work.
Catalog record
| CREATOR | Katherine Xie, Nitya Babbar, Vicky Chen, Yoanna Turura |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07990
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Multilingual Language Models for Code-Switched Input Data Katherine Xie, Nitya Babbar, Vicky Chen, Yoanna Turura Abstract 1 Abstract Code-switching, or alternating between languages within a single conversation, presents challenges for multilingual language models on NLP tasks. This research investigates if pre-training Multilingual BERT (mBERT) on code-switched datasets improves the model’s performance on critical NLP tasks such as part of speech tagging, sentiment analysis, named entity recognition, and language identification. We use a dataset of Spanglish tweets for pre-training and evaluate the pre-trained model against a baseline model. Our findings show that our pre-trained mBERT model outperforms or matches the baseline model in the given tasks, with the most significant improvements seen for parts of speech tagging. Additionally, our latent analysis uncovers more homogenous English and Spanish embeddings for language identification tasks, providing insights for future modeling work. This research highlights potential for adapting multilingual LMs for code-switched input data in order for advanced utility in globalized and multilingual contexts. Future work includes extending experiments to other language pairs, incorporating multiform data, and exploring methods for better understanding context-dependent code-switches. 2 Introduction In today’s increasingly globalized world, multilingualism is common, and code-switching—switching between languages within a single conversation—is particularly prevalent in informal communication. However, most current natural language processing (NLP) models are designed for monolingual text or cross-lingual tasks such as translation, leaving a significant gap in handling code-switched text. Code-switching presents unique challenges, including accurately capturing language transitions and seamlessly determining meaning across them. As demand grows for multilingual and mixed-language processing in applications such as chatbots and social media monitoring, addressing these challenges is critical, especially for multilingual and multicultural communities. Existing multilingual language models (LMs), such as mBERT (Multilingual BERT) and XLMR (Cross-lingual Language Model), are typically pre-trained on monolingual or translation-based datasets Conneau et al. 2020 , making them ill-suited for handling the unpredictable language transitions found in code-switched text. This limitation affects various NLP related tasks, often resulting in higher loss values and lower accuracies than monolingual inputs. Our study aims to fill this gap by pre-training mBERT, an existing multilingual LM, on a Spanglish code-switched dataset and focusing on evaluating the model with four different tasks: parts of speech tagging (POS), language identification (LID), named entity recognition (NER), and sentiment analysis (SA) tasks. In addition, we investigate the differences in latent space representations generated by the model before and after pre-training to interpret how the model’s understanding of code-switched input data changes. By improving the model’s ability to capture language transitions and generate high-quality outputs from code-switched text, we aim to contribute not only to technical advances in NLP but also to provide better natural language understanding in multilingual communities. 3 Related Work Research in code-switching within the field of natural language processing has gained traction in recent years due to the challenges of handling multilingual data, including capturing linguistic transitions, syntactic patterns, and semantic coherence across different languages. Although not specifically designed for code-switching, a paper by Ammar et al. 2016 laid some groundwork for multilingual models by introducing embeddings that allow words from different languages to be represented in a shared vector space. This has led to the development of multilingual models like mBERT and XLMR, which in recent years have been applied towards cross-lingual tasks. A study by Aguilar et al. 2020 developed the LinCE benchmark to provide standardized evaluation metrics for code-switching across various tasks and language pairs, aiming for consistency in evaluating the performance of multilingual models on code-switched data. A study by Liu et al. 2019 introduced RoBERTa, an optimized variant of BERT, which refined BERT’s training process to improve performance without task-specific fine-tuning. RoBERTa removed the Next Sentence Prediction (NSP) objective included in the original BERT model, but was found to add limited value to downstream tasks. The researchers also implemented dynamic masking, where the tokens selected for masking were randomized in each training iteration, enhancing the model’s ability to generalize instead of overfitting to the fixed masked word positions. Although modifications were made on BERT, these changes have potential to improve the performance of mBERT on code-switched data to better capture complex linguistic patterns across languages. In 2018, Bhat et al. 2018 contributed to research on code-switched language modeling by proposing models specifically tailored for Hindi-English code-switched data, using LSTM-based approaches, which proved effective for sequence labeling tasks but were limited in other abilities. There has also been research related to latent space representations created by models from code-switched data. A study by Wu et al. 2015 uses Latent Language Space Models (LLSMs) to focus on accurately identifying language change points within code-switched data, while Asnani and Pawar 2018 use Latent Dirichlet Allocation (LDA), which is an extension of Probabilistic Latent Semantic Analysis (PLSA), to focus on topic modeling within code-switched text. A paper by De Leon et al. 2024 includes interpretability analysis on latent space embeddings generated by pre-trained models on code-switched text. Outside of the field of natural language processing, many research papers have focused on the linguistics of code-switching. Work by Lipski 2008 and Dussias et al. 2014 focused on exploring grammatical rules for code-switching, and where language transitions most frequently occur within a sentence. These results can be used to test a model’s understanding of grammatical rules related to code-switching. Our work builds upon this existing research by directly addressing the limitations of current multilingual models in handling NLP tasks in code-switched contexts. We leverage the mBERT model, fine-tune it on code-switched datasets, and conduct experiments to analyze how the new model understands code-switched data. We aim to improve upon the model’s ability to produce accurate output based on multilingual sentences, and gain some intuition as to how the improved model captures their semantic meaning and whether this aligns with generally accepted grammatical rules for code-switching. 4 Evaluation We will evaluate our pre-trained model for its performance on token classification and sentence classification tasks, as measured by various metrics. The model’s performance will be benchmarked against a baseline mBERT transformer model without specific code-switching pre-training. We will use the LinCE benchmark Aguilar et al. 2020 , which provides annotated datasets for tasks such as POS tagging, LID, NER, and SA tasks across several language pairs. We will use the provided Spanglish datasets for each of these tasks that contain short code-switched tweets. Performance of our model will be evaluated using LinCE’s accuracy, F1 score, Precision, and Recall metrics for each task. Additionally, for the SA and LID tasks, we use latent semantic analysis to observe how the latent space embeddings of code-switched input data differ before and after pre-training. We compress the representations to two dimensions to allow for visual analysis through graphing. Finally, for the POS tagging task, we analyze the parts of speech on which code-switching most frequently occurs within a sentence. We then compare this data to linguistic literature such as Lipski 2008 and Dussias et al. 2014 to determine whether our model seems to accurately capture the grammar rules of code-switching. In the sections below, we outline each of the tasks we used to evaluate our model. Evaluating LMs on these four tasks with code-switched text can help to determine the model’s ability to process multilingual data and