New Evaluation Paradigm for Lexical Simplification

Jipeng Qiang, Minjiang Huang, Yi Zhu, Yunhao Yuan, Chaowei Zhang, Xiaoye Ouyang · 2025

Read New Evaluation Paradigm for Lexical Simplification by Jipeng Qiang, Minjiang Huang, Yi Zhu, Yunhao Yuan, Chaowei Zhang, Xiaoye Ouyang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15268.txt.

Introduction

Jipeng Qiang and co-authors rethink how to grade word simplification in a 2025 paper. Software that swaps hard words for easy ones is usually built and tested in three separate steps, each with its own data set.

Large language models skip the steps and rewrite whole sentences from one prompt. Old test sets, built around single hard words, cannot judge those rewrites fairly. The authors build an all in one data set by mixing machine proposals with human judgment, so every hard word in a sentence is covered. They also try prompting tricks and a team of models that mimics the three classic steps. The team approach beat the existing baselines by a clear margin.

Catalog record

CREATORJipeng Qiang, Minjiang Huang, Yi Zhu, Yunhao Yuan, Chaowei Zhang, Xiaoye Ouyang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-15268

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

New Evaluation Paradigm for Lexical Simplification Jipeng Qiang, Minjiang Huang, Yi Zhu, Yunhao Yuan, Chaowei Zhang, Xiaoye Ouyang Abstract Lexical Simplification (LS) methods use a three-step pipeline: complex word identification, substitute generation, and substitute ranking, each with separate evaluation datasets. We found large language models (LLMs) can simplify sentences directly with a single prompt, bypassing the traditional pipeline. However, existing LS datasets are not suitable for evaluating these LLM-generated simplified sentences, as they focus on providing substitutes for single complex words without identifying all complex words in a sentence. To address this gap, we propose a new annotation method for constructing an all-in-one LS dataset through human-machine collaboration. Automated methods generate a pool of potential substitutes, which human annotators then assess, suggesting additional alternatives as needed. Additionally, we explore LLM-based methods with single prompts, in-context learning, and chain-of-thought techniques. We introduce a multi-LLMs collaboration approach to simulate each step of the LS task. Experimental results demonstrate that LS based on multi-LLMs approaches significantly outperforms existing baselines. 1 Introduction The aim of Lexical Simplification (LS) is to substitute complex words within sentences with simpler alternatives, thereby enhancing reading comprehension for a diverse array of readers, including non-native speakers ( Paetzold and Specia, 2016 ) and individuals with cognitive impairments ( Saggion, 2017 ) . LS method is commonly framed as a pipeline of three main steps: Complex Word Identification (CWI), Substitute Generation (SG), and Substitute Ranking (SR) ( Paetzold and Specia, 2017b ; Qiang et al., 2021a ) . The three steps are treated as different independent tasks, and they have their own evaluation datasets. The CWI step identifies complex words or phrases in a text, which can be treated as classification or sequence labeling task Yimam et al. (2018) ; Paetzold and Specia (2017a) ; Gooding and Kochmar (2019) ; Qiang et al. (2021a) . The SG step generates many simpler alternatives for complex words by Linguistic-based methods Pavlick and Callison-Burch (2016) ; Maddela and Xu (2018) , word embedding-based methods Glavaš and Štajner (2015) ; Paetzold and Specia (2017a) , pretrained language model-based methods Qiang et al. (2020) ; Liu et al. (2023) , or Large language model (LLM) based methods Sheang and Saggion (2023) . The SR step ranks the generated substitutes of the SG step by considering the context and the complex word Paetzold and Specia (2017a) ; Qiang et al. (2021b) . This step typically combines multiple features for ranking, including word frequency, meaning preserved, contextual fluency, and so on. Figure 1: Flowchart of current LS methods and LLM-based LS method. Current LS methods require not only three models for three steps: complex word identification, substitute generation, and substitute ranking, but also iterative simplification of each complex word. We found that the LLM-based method only needs a single prompting to complete the task. Recently, we have found that LLM can directly output simplified sentences that satisfy the LS requirements as long as the appropriate prompt template is designed. As shown in Figure 1 , after feeding the original sentence "John composed these verses" into LLMs, two simplified sentences generated from LLMs (GPT-3.5and GPT-4o-mini) ("John wrote these poems" and "John composed these poems"), which one is better?. LLM-based method shows its advantage without a pipeline of three steps. However, it brings a new problem: Can the existing LS datasets evaluate this type of simplified sentence? CWI datasets ( Yimam et al., 2018 ; Gooding and Kochmar, 2018 ) only provide all the complex words for each sentence. Each instance in LS datasets Paetzold and Specia (2017a) ; Qiang et al. (2021a) consists of a sentence, a complex word, and a list of gold candidates. These datasets only focus on providing many substitutes for one complex word and do not identify all complex words within a sentence. We cannot use the existing datasets to evaluate the LS performance of different LLMs. The evaluation of such methods should consider the following two aspects: (1) Methods that correctly simplify a greater number of complex words should exhibit better performance. (2) Considering that the difficulty of complex words varies, methods that simplify more challenging complex words should demonstrate superior performance. Constructing a dataset that can directly evaluate this type of simplification will open up a new evaluation paradigm for LS task. In this paper, we will propose a novel annotation method to construct an all-in-one style LS Dataset based on human-computer collaboration. Considering that existing CWI datasets have already annotated complex words, we built LS datasets based on the CWI datasets, thus saving the workload of annotating complex words. Given a complex word, we did not manually annotate the substitutes; instead, we used a human-computer interaction approach to generate the substitutes, since multiple automated LS methods can swiftly generate a vast pool of potential substitutes, alleviating the burden on human annotators. Subsequently, human annotators assess the suitability of these alternatives. To address the LS task, we explore not only LLM-based methods with a single prompt but also integrate in-context learning and chain-of-thought techniques. To further improve the performance of LLM-based methods, we propose a novel method LLM via multi-LLMs collaboration. This work utilizes multi-LLM collaboration to simulate each step (CWI, SG, and SR) of LS task. Experimental results show that LLM-based methods outperform existing multi-step approaches, and the multi-LLMs-based method significantly outperforms LLM-based method. 2 Related Work All existing LS systems are framed as a pipeline of three or four steps: Complex Word Identification, Substitute Generation, Substitute Selection (SS, optional), and Substitute Ranking (SR) Paetzold and Specia (2017b) ; Qiang et al. (2021a) . We will cover the following three areas (CWI, SG, and SR). CWI. Early methods Paetzold and Specia (2017b) used manual features and traditional machine learning. Later, the research included ensemble-based classifier methods, and the field expanded to multilingual CWI with notable milestones like CWI 2018 shared task Yimam et al. (2018) ; Gooding and Kochmar (2018) . The rise of deep learning introduced neural network models that outperformed earlier techniques, particularly with the use of word embeddings. Recent advances, such as BERT-based models Qiang et al. (2021a) , further improved performance by capturing context-dependent meanings. Hybrid models combining various methods have also shown promise North et al. (2024) . Standardized datasets and evaluation metrics, such as those from the Lexical Complexity Prediction 2021 Shared Task Shardlow et al. (2021) , have been crucial for progress. The field continues to evolve, aiming to create more robust cross-lingual models and adaptive systems, enhancing text accessibility and readability for diverse applications North et al. (2023) . SG. Early methods, e.g., linguistic-based methods Pavlick and Callison-Burch (2016) ; Maddela and Xu (2018) and word embedding-based Methods Glavaš and Štajner (2015) ; Paetzold and Specia (2017a) , leverage linguistic resources or word embedding models to generate simpler substitutes. Pretrained language models have significantly advanced the field of lexical simplification Sheang et al. (2022) . Qiang et al. Qiang et al. (2020) and Liu et al. Liu et al. (2023) adopted pretrained language models to generate context-aware substitutes. These approaches leverage the contextual understanding of PLMs, resulting in more accurate and contextually appropriate simplifications. The advent of large language models (LLMs) such as GPT-3 and ChatGPT has opened new possibilities for substitute generation.

Read the complete text