Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain Setups
Răzvan-Alexandru Smădu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel · 2024
Read Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain Setups by Răzvan-Alexandru Smădu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01706.txt.
Description
Abstract Complex Word Identification (CWI) is an essential step in the lexical simplification task and has recently become a task on its own. Some variations of this binary classification task have emerged, such as lexical complexity prediction (LCP) and complexity evaluation of multi-word expressions (MWE).
Catalog record
| CREATOR | Răzvan-Alexandru Smădu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01706
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain Setups Răzvan-Alexandru Smădu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel Abstract Complex Word Identification (CWI) is an essential step in the lexical simplification task and has recently become a task on its own. Some variations of this binary classification task have emerged, such as lexical complexity prediction (LCP) and complexity evaluation of multi-word expressions (MWE). Large language models (LLMs) recently became popular in the Natural Language Processing community because of their versatility and capability to solve unseen tasks in zero/few-shot settings. Our work investigates LLM usage, specifically open-source models such as Llama 2, Llama 3, and Vicuna v1.5, and closed-source, such as ChatGPT-3.5-turbo and GPT-4o, in the CWI, LCP, and MWE settings. We evaluate zero-shot, few-shot, and fine-tuning settings and show that LLMs struggle in certain conditions or achieve comparable results against existing methods. In addition, we provide some views on meta-learning combined with prompt learning. In the end, we conclude that the current state of LLMs cannot or barely outperform existing methods, which are usually much smaller. 1 Introduction Complex word identification (CWI) aims to determine whether words or phrases are difficult for a target group of readers to understand. In the lexical simplification task, which targets replacing complex words and expressions with simplified alternatives North et al. (2023a) , CWI is the first step, and it was treated as part of this task until 2012 when it became a standalone task Shardlow (2013) . CWI was initially addressed as a binary classification task Paetzold and Specia (2016) , identifying whether a word is complex in a given sentence. When the task became more popular North et al. (2023b) , it was extended to the continuous domain as Lexical Complexity Prediction (LCP, also referred to as the probabilistic classification for CWI) Yimam et al. (2018) , addressing multilingual and multidomain settings. Then, it was extended to multi-word expressions Shardlow et al. (2021) . Recently, new datasets started to emerge in various languages and domains Ortiz Zambrano and Montejo-Ráez (2021) ; Venugopal et al. (2022) ; Ilgen and Biemann (2023) ; Zambrano et al. (2023) . Previous approaches to CWI ranged from using Support Vector Machines S.P et al. (2016) to deep neural networks based on Bidirectional Representation from Encoder Transformers Pan et al. (2021) , multi-task learning with domain adaptation Zaharia et al. (2022) , and sequence modeling Gooding and Kochmar (2019) . With the recent breakthrough in large language models (LLMs), particularly the work of OpenAI with Generative Pre-trained Transformer (GPT) models Radford et al. (2019) ; Brown et al. (2020) , natural language processing has seen a significant leap. These models have shown the potential to improve performances on various tasks as we scale up the model size and the amount of training data. The announcement of ChatGPT 1 1 1 https://openai.com/blog/chatgpt and its conversational capabilities has sparked a race in developing and fine-tuning new models for general purpose and domain-specific applications using PaLM Anil et al. (2023) , LLaMA Touvron et al. (2023) ; Dubey et al. (2024) , Orca Mitra et al. (2023) , Mistral Jiang et al. (2023) , GPT-4 OpenAI et al. (2023) , GPT-4o 2 2 2 https://openai.com/index/hello-gpt-4o/ , and many others. Our work aims to provide the current state of LLMs in addressing CWI and LCP compared against state-of-the-art approaches. We focus on evaluating open-source (pre-trained Llama 2 Touvron et al. (2023) , Llama 3 Dubey et al. (2024) , Vicuna Zheng et al. (2024) ) and OpenAI’s close-source ChatGPT-3.5-turbo and GPT-4o. We summarize the contributions as follows: • We evaluate LLMs in binary (discrete set of labels) and probabilistic classification (continuous space labels) on multidomain and multilingual corpora. • We employ various techniques for prompting and fine-tuning. • We show that LLMs struggle to address CWI and LCP tasks; however, in limited instances, they can achieve similar results with other, more lightweight approaches. • In the end, we analyze and provide an insight into where the models struggle. 2 Related Work 2.1 Complex Word Identification CWI was previously addressed using straightforward baseline models based on feature engineering. For example, Aroyehun et al. (2018) compared CNN-based models with various feature engineering methods based on tree ensembles and features, such as inverse term frequency, parts-of-speech tagging, WordNet, and word2vec, achieving comparable results. Finnimore et al. (2019) proposed mono and cross-lingual models based on simple features and logistic regression, achieving similar results to more complex, language-specific state-of-the-art models. Zaharia et al. (2020) employed zero and few-shot learning techniques, along with Transformers and Recurrent Neural Networks, in a multilingual setting. Gooding and Kochmar (2019) considered CWI a sequential task, using a bidirectional LSTM with word embeddings, character-level representations, and a language modeling objective to learn the complexity of words given the context. Other methods improved performances, employing graph-based Ehara (2019) , domain adaptation Zaharia et al. (2022) , and transformer-based models Pan et al. (2021) ; Cheng Sheang et al. (2022) . 2.2 Large Language Models Recently, LLMs were successfully utilized in various generative tasks Pu et al. (2023) ; Chen et al. (2021) . The new paradigm in solving other non-generative tasks is based on prompting pre-trained language models to perform the prediction task Liu et al. (2023a) ; Sun et al. (2023) . Fine-tuning models on instructions showed improved results in zero-shot settings, especially on unseen tasks Wei et al. (2022a) . Prompt-based methods such as the use of demonstrations Min et al. (2022) , intermediate reasoning steps by breaking down complex tasks into simpler subtasks (also known as a chain of thought) Wei et al. (2022b) , and using LLMs to optimize their prompts Zhou et al. (2023) made zero-shot inference much more appealing due to reduced costs and more efficient than fine-tuning LLMs. 2.3 Prompt Tuning and Meta-Learning Fine-tuning usually requires high computational costs, especially in the context of LLMs, to achieve good performances. Therefore, there was the need to find alternatives. Prompt Tuning Lester et al. (2021) is a soft-prompting technique for adapting a large language model for a custom task without training its parameters. While successful, prompt tuning falls short when applied in the few-shot learning regime, leading to a combination with meta-learning. MetaPrompting Hou et al. (2022) utilizes the meta-learning algorithms FOMAML Finn et al. (2017) and Reptile Nichol et al. (2018) to obtain optimized initialization embeddings. One shortcoming of this approach is the requirement for supervised training data during meta-learning, which is alleviated in other works Pan et al. (2023) ; Huang et al. (2023) by generating the meta-training tasks in an unsupervised or self-supervised manner. Additionally, other works explored the use of an adaptable gradient regulating function Li et al. (2023) and domain-adversarial neural networks Fang et al. (2023) , both techniques used to increase model generalization. 3 Method 3.1 Problem Formulation in the Pre-LLM Era Word complexity can be defined as absolute and relative North et al. (2023b) . Absolute complexity is determined by the objective linguistic properties (e.g., semantic, morphological, phonological). In contrast, relative complexity is related to the subjective speaker’s point of view (e.g., familiarity with sound and meaning, culture, and context). In this work, we evaluate the relative