Improving Estonian Text Simplification through Pretrained Language Models and Custom Datasets

Eduard Barbu, Meeri-Ly Muru, Sten Marcus Malva · 2025

Read Improving Estonian Text Simplification through Pretrained Language Models and Custom Datasets by Eduard Barbu, Meeri-Ly Muru, Sten Marcus Malva (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15624.txt.

Introduction

Eduard Barbu and co-authors report a language technology study, published in 2025, on making Estonian text easier to read. Estonian has few resources for this task, so the team first built a new dataset. They combined texts simplified by hand with versions made by a large language model, giving both systems the same material to learn from.

Two approaches were compared: a translation style neural system and a fine-tuned large language model called LLaMA. Judges rated the language model higher on grammar, ease of reading, and keeping the original meaning. The team also released its dataset, training scripts, and tests, so others can adapt the method to other low resource languages.

Catalog record

CREATOREduard Barbu, Meeri-Ly Muru, Sten Marcus Malva
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-15624

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Estonian Text Simplification through Pretrained Language Models and Custom Datasets Eduard Barbu, Meeri-Ly Muru, Sten Marcus Malva Abstract This study introduces an approach to Estonian text simplification using two model architectures: a neural machine translation model and a fine-tuned large language model (LLaMA). Given the limited resources for Estonian, we developed a new dataset, the Estonian Simplification Dataset, combining translated data and GPT-4.0-generated simplifications. We benchmarked OpenNMT, a neural machine translation model that frames text simplification as a translation task, and fine-tuned the LLaMA model on our dataset to tailor it specifically for Estonian simplification. Manual evaluations on the test set show that the LLaMA model consistently outperforms OpenNMT in readability, grammaticality, and meaning preservation. These findings underscore the potential of large language models for low-resource languages and provide a basis for further research in Estonian text simplification. K eywords text simplification \cdot LLM \cdot machine translation 1 Introduction and Related work Text simplification involves converting complex text into simpler forms while preserving the original meaning. This is typically done by splitting the text into sentences, shortening sentence length, simplifying vocabulary, and simplifying grammatical structures. It is essential for improving accessibility for readers with lower language proficiency, cognitive impairments, or for those learning English as a second language. Additionally, it has applications in educational tools, assistive technologies, and content tailored for broader audiences. Automatic text simplification (ATS) leverages natural language processing (NLP) algorithms to enhance readability while maintaining the core information. Most research in text simplification focuses on the English language. Comprehensive reviews of work in ATS focusing on the types of simplification, tools and datasets can be found in Saggion 2017 ; Espinosa-Zaragoza et al. 2023 ; Shardlow 2014 . Over the years, various projects have applied ATS to different target groups, leading to the development of various tools. Notable examples include the HAPPI project Devlin and Unthank 2006 , aimed at individuals with aphasia, the PorSimples project Aluisio et al. 2010 for people with low literacy, and the FIRST project Barbu et al. 2015 , which focused on individuals with autism and cognitive challenges. Most research in ATS has not specified a target group, instead focusing on different simplification phenomena. As in other areas of NLP the best architectures for text simplification work a based on deep neural networks ( Zhao et al. 2018 ; Anjum and Lieberum 2023 ) Pretrained LLMs, such as GPT OpenAI 2023 , LLaMA Touvron et al. 2023 , and T5 Raffel et al. 2020a , have become powerful tools for various natural language processing tasks, including text simplification. These models are trained on massive amounts of text data, enabling them to understand and generate human-like language. When fine-tuned for text simplification, LLMs can simplify complex sentences by rephrasing them, reducing sentence length, and substituting difficult words with simpler alternatives while maintaining the original meaning. Estonian ATS has only been explored at the bachelor thesis level, with two attempts: one focused on lexical simplification using the Estonian WordNet and cosine similarity, and the other on syntactic simplification, guided by English structures. In this study, we benchmark two models for Estonian text simplification. The first model is based on neural machine translation, treating ATS as a translation task. The second model involves fine-tuning the LLaMA language model on the Estonian Simplification Dataset. While standard metrics such as BLEU and SARI are often used to evaluate text simplification models, they do not capture essential aspects like readability and meaning preservation. This study combines these metrics with manual evaluations to provide a comprehensive assessment The structure of the paper is as follows: Section 2 describes the creation of the Estonian Simplification Dataset. Section 3 replicates an English-language experiment Zhang and Lapata 2017 and provides the rationale for selecting OpenNMT as the baseline for Estonian ATS benchmarking. Section 4 covers the fine-tuning of LLaMA on the Estonian dataset with optimization using the Unsloth library. The final two sections present the evaluation results for both models and present the conclusions. 2 Building the Estonian Simplification Dataset In this section, we outline the process of constructing the Estonian Simplification Dataset. The primary goal is to create a sufficiently large dataset to effectively train deep neural network models. Given the required scale and the lack of any existing simplification dataset for Estonian, it is clear that manually compiling this dataset would be impractical. Where possible, the dataset is aligned with simplified English sentences, and we also aim to establish guidelines to capture linguistic phenomena specific to Estonian, suitable for simplification. The first source for our dataset is the Turk corpus Xu et al. 2016 , which contains 2,359 original Wikipedia sentences, each paired with eight simplified versions created via Amazon Mechanical Turk. This corpus provides a variety of simplification styles, including lexical substitution and sentence rephrasing. Since it is widely used in text simplification research, we manually translated relevant portions into Estonian, focusing on sentences that preserved a simplification relationship. The second source is Wikipedia Data Set 2.0 Kauchak 2013 , which contains 167,689 aligned Wikipedia sentences. Small samples from this dataset were translated using machine translation and then corrected by Estonian annotators. However, the dataset ultimately proved unreliable, as many sentences lack coherence. To ensure consistency and reproducibility of simplification techniques, we developed the Estonian Simplification Guidelines, inspired in part by the Newsela Corpus Xu et al. 2015 . The Newsela Corpus includes 1,130 news articles, each rewritten at four different levels to cater to readers of varying abilities, with professional editors ensuring quality and readability. Although the corpus is not freely available, it can be accessed for academic or research purposes. Our Estonian Simplification Guidelines consist of a detailed 10-page document outlining specific simplification techniques tailored to Estonian. These guidelines focus on three main areas: • Grammatical complexity: Simplifying cases, tenses, and grammatical moods. • Syntactic complexity: Reducing sentence length, splitting long sentences, and rearranging word order for clarity. • Lexical simplification: Replacing complex words with simpler synonyms or broader terms to improve comprehension. Following the approach used in English-language research, the primary source for Estonian simplification data is Estonian Wikipedia. Articles are downloaded from the latest dumps, split into individual sentences, and only sentences with more than 15 words are selected for simplification. To efficiently generate many simplified sentences, we employed GPT-4.0 OpenAI 2023 , a state-of-the-art language model with a high ranking on the Chatbot Arena LLM Leaderboard. Although generating data with GPT-4.0 incurs costs, they are significantly lower than those associated with human simplification. The model, pretrained on extensive datasets, can be fine-tuned to adopt specific personas, enabling it to emulate professional tones and styles tailored to various roles, such as technical support or educational tasks. Studies have shown that persona-based prompting techniques outperform other prompting methods and are widely adopted in practice Pataranutaporn et al. 2021 ; Wang et al. 2024 . In our research, we developed a variety of prompting templates incorporating different simplification strategies. These templates were tested across various instances of GPT-4.0. The initial templates did not incorporate any persona style, while later versions introduced persona agents focused on syntactic and lexical simplification. After experimentation, we found that the best results were achieved by first applying lexical simplification, followed by

Read the complete text