Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contexts

Frances Laureano De Leon, Harish Tayyar Madabushi, Mark G. Lee · 2025

Read Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contexts by Frances Laureano De Leon, Harish Tayyar Madabushi, Mark G. Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20051.txt.

Introduction

Frances Laureano De Leon, Harish Tayyar Madabushi, and Mark Lee test how well large language models handle multiword expressions, in a 2025 paper. Such expressions, like idioms, mean more than their words say and can switch between literal and figurative readings. The team checked the models in English, Portuguese, and Galician, and built a new data set where sentences switch languages mid-stream. A new task probed the same skill from a fresh angle.

Even the newest models, GPT-4 among them, failed to beat a modest older baseline at spotting and interpreting the expressions. Scores sagged most on the novel tasks and on the less common languages. The authors conclude that ambiguous multiword expressions remain a weak spot for these models. Memorized patterns seem to carry the models where they do succeed.

Catalog record

CREATORFrances Laureano De Leon, Harish Tayyar Madabushi, Mark G. Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20051

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contexts Thanks: Citation : Authors. Title. Pages…. DOI:000000/11111. Frances Laureano De Leon, Harish Tayyar Madabushi, Mark G. Lee Abstract Multiword expressions, characterised by non-compositional meanings and syntactic irregularities, are an example of nuanced language. These expressions can be used literally or idiomatically, leading to significant changes in meaning. While large language models have demonstrated strong performance across many tasks, their ability to handle such linguistic subtleties remains uncertain. Therefore, this study evaluates how state-of-the-art language models process the ambiguity of potentially idiomatic multiword expressions, particularly in contexts that are less frequent, where models are less likely to rely on memorisation. By evaluating models across in Portuguese and Galician, in addition to English, and using a novel code-switched dataset and a novel task, we find that large language models, despite their strengths, struggle with nuanced language. In particular, we find that the latest models, including GPT-4, fail to outperform the xlm-roBERTa-base baselines in both detection and semantic tasks, with especially poor performance on the novel tasks we introduce, despite its similarity to existing tasks. Overall, our results demonstrate that multiword expressions, especially those which are ambiguous, continue to be a challenge to models. K eywords Multiword expressions \cdot Code-switching \cdot Large language models 1 Introduction Multiword expressions (MWEs) present significant challenges in Natural Language Processing (NLP) due to their linguistic complexities. MWEs consist of multiple words that convey a specific meaning, which may be non-compositional and exhibit syntactic irregularities [ 1 , 2 ] . These expressions are widespread across languages and domains, from the arts to the sciences [ 3 ] . The number of MWEs in a speaker’s lexicon is estimated to be comparable to that of single words [ 4 ] . MWEs are often rooted in cultural context and communication rather than a straightforward lexical meaning, making their interpretation in NLP systems more challenging, particularly in idiomatic contexts [ 5 ] . Additionally, MWEs can be ambiguous, functioning either idiomatically or literally, which can cause potential confusion [ 6 ] . For example, the English MWE break the ice can refer to its literal meaning or describe making unfamiliar individuals feel more at ease with one another. Such expressions, known as potentially idiomatic expressions (PIEs) [ 7 ] , require an understanding of both literal and figurative meanings. Given the idiosyncratic nature of MWEs, it is essential that state-of-the-art NLP models accurately capture their meanings. Large language models (LLMs) have rapidly become the dominant approach in NLP, demonstrating strong language understanding and generation capabilities across a wide range of tasks in zero-shot and few-shot settings [ 8 ] . However, despite their strong performance, LLMs have limitations, including the generation of hallucinations—fluent but inaccurate or fabricated text—and a reliance on statistical co-occurrence patterns from pre-training data [ 9 , 10 ] . This study, therefore, aims to assess whether LLMs can genuinely understand nuanced language, using MWEs as a means of evaluation. MWEs are particularly relevant because their meanings can be either idiomatic or literal, requiring an understanding of both social and linguistic aspects of language. Since MWEs exist in all languages and vary in frequency and usage, their interpretation may be further complicated in multilingual contexts. One such challenge arises in code-switching (CS), where speakers alternate between languages within a conversation or sentence [ 11 , 12 ] . Code-switching is common in multilingual communities and presents additional difficulties for NLP systems, as MWEs may be split across languages. To evaluate how LLMs handle this complexity, we use data in English, Portuguese, and Galician, as well as CS examples. By including both high-resource and low-resource languages alongside CS data, we aim to assess whether models rely primarily on statistical patterns in training data or if they can generalise effectively across multilingual and code-switched contexts. We conduct experiments using both open-source and closed-source LLMs to examine how these models handle MWEs across different languages and contexts. Specifically, we investigate: • the ability of LLMs to detect whether MWEs are used as idioms or literally, • the extent to which LLMs effectively capture the non-compositional meanings of MWEs, • the ability of LLMs to acquire information from the prompt, and • the relationship between LLM size and performance on text containing MWEs with non-compositional meaning. Our experiments reveal that, despite their proficiency in many other tasks, LLMs continue to struggle with processing MWEs. This gap highlights the limitations of these models compared to fine-tuned pre-trained language models (PLMs) such as xlm-roBERTa, which are much smaller models and contain many fewer parameters. While some results demonstrate handling of MWEs with non-compositional meanings in English comparable to fine-tuned PLMs, the same cannot be said for other languages. Our experiments highlight the limitations of current models and emphasise the need for careful consideration when applying them in multilingual contexts, particularly with nuanced language. Section 2 reviews the relevant literature on MWEs. In section 3 , we describe the datasets used in this study and outline the creation of the synthetic CS dataset. Section 4 provides a detailed account of our experimental setup, while section 5 presents and discusses our findings. Finally, we conclude with a summary of our results and recommendations for future research in section 6 . 1.1 Contributions We carry out experiments to assess the ability of generative models to detect and interpret MWEs in both idiomatic and compositional contexts. MWEs are used for evaluation due to their lower frequency, reducing the likelihood of LLMs relying on memorisation. To further investigate whether LLMs use memorisation in MWE interpretation, we create a novel CS test dataset with examples in three language pairs—Galician-Spanish, English-Spanish, and Portuguese-Spanish—and make it publicly available. To address the research questions above, we evaluate models not only on the standard task of detecting multiword expressions (Section 4.3 ) but also on novel tasks assessing the semantic interpretation of MWEs (Section 4.5 ). This includes a combination of synthetic MWEs (Section 4.4 ) and a new code-switched dataset (Section 3.1 ). By introducing these datasets, tasks, and settings, we aim to reduce the likelihood of model memorisation and assess their ability to interpret MWEs, particularly in non-compositional contexts. Our results demonstrate that LLMs are not fully capable of detecting and representing the semantics of MWEs. Furthermore, they indicate that prompt-based learning, by providing information in the prompt, is not sufficient for the model to handle unseen vocabulary, in challenging linguistic tasks. 2 Related work Studies have shown that popular transformer models generally struggle to effectively handle figurative language. Research on encoder-only models shows that PLMs do not adequately capture and represent the meanings of idiomatic expressions [ 13 ] . Additionally, generative models like GPT-2 have been found to perform poorly in handling figurative language without the use of mitigation strategies [ 14 ] . [ 15 ] reveal that transformer models capture MWE semantics inconsistently, and find that they are reliant on surface patterns. However, research has shown that encoder-only PLMs can effectively represent MWEs when fine-tuned with appropriate data [ 16 ] . Studies exploring how generative models handle figurative language are few, especially studies focusing on idioms. [ 8 ] test BERT, and RoBERTa,

Read the complete text