Towards Automated Lexicography: Generating and Evaluating Definitions for Learner’s Dictionaries
Yusuke Ide Adam Nohejl Joshua Tanner, Hitomi Yanaka Christopher Lindsay Taro Watanabe · 2026
Read Towards Automated Lexicography: Generating and Evaluating Definitions for Learner’s Dictionaries by Yusuke Ide Adam Nohejl Joshua Tanner, Hitomi Yanaka Christopher Lindsay Taro Watanabe (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01842.txt.
Introduction
Writing dictionary definitions simple enough for language learners is skilled, expensive work. Yusuke Ide and five co-authors study generating such definitions automatically with language models, and first build a trustworthy way to judge the output.
The team worked with a professional dictionary maker. They assembled a reference dataset of Japanese definitions. They also proposed scoring rules. A language model applies the rules as judge. Its scores agreed with human judges about as well as the humans agreed with each other. Their generation method then simplifies its own definitions in rounds. The vocabulary stays easy. The resulting definitions scored highly on the new rules. Their words stayed simple. The work is a step toward automated dictionary writing that respects how learners actually read.
Catalog record
| CREATOR | Yusuke Ide Adam Nohejl Joshua Tanner, Hitomi Yanaka Christopher Lindsay Taro Watanabe |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-01842
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Automated Lexicography: Generating and Evaluating Definitions for Learner’s Dictionaries Yusuke Ide Adam Nohejl Joshua Tanner, Hitomi Yanaka Christopher Lindsay Taro Watanabe Abstract We study dictionary definition generation (DDG), i.e., the generation of non-contextualized definitions for given headwords. Dictionary definitions are an essential resource for learning word senses, but manually creating them is costly, which motivates us to automate the process. Specifically, we address learner’s dictionary definition generation (LDDG), where definitions should consist of simple words. First, we introduce a reliable evaluation approach for DDG, based on our new evaluation criteria and powered by an LLM-as-a-judge. To provide reference definitions for the evaluation, we also construct a Japanese dataset in collaboration with a professional lexicographer. Validation results demonstrate that our evaluation approach agrees reasonably well with human annotators. Second, we propose an LDDG approach via iterative simplification with an LLM. Experimental results indicate that definitions generated by our approach achieve high scores on our criteria while maintaining lexical simplicity. 1 1 1 Our code and dataset will be publicly available upon acceptance. 1 Introduction Dictionary definitions are an essential resource for learning and understanding word senses. They can also be used as resources for various tasks in natural language processing (NLP), e.g., word sense disambiguation. Creating definitions for myriads of words in a dictionary, however, incurs considerable cost. To address this issue, we study dictionary definition generation (DDG), i.e., the generation of definitions for given headwords without context. Specifically, we focus on definition generation for learner’s dictionaries (LDs), where the definitions should consist of simple words, referring to the task as learner’s dictionary definition generation (LDDG). LDs constitute a major category of dictionaries. For English, six major dictionary publishers, such as Oxford University Press, have produced LDs 13 . LDs help learners develop language capabilities, particularly in production 37 . However, many languages, e.g., Spanish and Japanese, lack high-quality LDs despite a large number of learners. Recent LLM-based systems have been shown to be effective on tasks traditionally handled by human experts, e.g., Wikipedia-style article writing 38 and scientific paper writing 25 . They can also simplify a wide variety of texts to improve user comprehension 12 , suggesting the feasibility of automatically generating correct and plain definitions that rival those written by humans. To evaluate generated definitions, previous studies have relied either on costly human evaluations or on metrics that do not fully account for the multi-dimensional nature of DDG evaluation. In this paper, we propose a reproducible and fine-grained evaluation approach based on four newly designed criteria—truthfulness, coverage, sense specificity, and guideline compliance. We evaluate them using an LLM-as-a-judge approach and a new dataset, the Dictionary Definition Dataset for Japanese (D3J), created in collaboration with a professional lexicographer. We validate the LLM-as-a-judge approach and show that it agrees reasonably well with human annotators, with a Kendall’s \tau of 0.631 on the average score across the four criteria. We also observe that it achieves higher agreement with humans on the criteria we use than conventional metrics, including BLEU 34 , which have been used for previous studies. We then propose a simple LDDG approach that combines few-shot prompting and iterative simplification, which uses an LLM to remove complex words one by one from definitions. Results by our evaluation framework indicate that our approach achieves high performance, surpassing Wiktionary on multiple criteria, while maintaining lexical simplicity. Task Example (Input \rightarrow Output) Contextualized def. gen. ( “bank” , “… the bank of the Nile …” ) \rightarrow “the side of a river, canal, etc.” Dictionary def. gen. (Our focus) “bank” \rightarrow [ “an organization that provides various financial services …” , “the side of a river, canal, etc.” , …] Table 1: Types of definition generation tasks. Example definitions are from 23 . 2 Related Work 2.1 Overview of Definition Generation Definition modeling and definition generation Studies on the generation of definitions have been referred to as definition modeling or definition generation , often without a clear distinction. In contrast, we differentiate between them based on the purpose of the study. Definition modeling is the task of generating definitions for given headwords and their embeddings in order to evaluate the embeddings 32 . The seminal work by 32 studied definitions for static embeddings. Subsequent studies, such as 10 , generated definitions that account for the context and senses of target words. Another line of studies has worked on generating definitions, primarily aiming to describe the meanings of unfamiliar words or phrases, with or without their embeddings. We refer to this task as definition generation, following previous studies, such as 19 ; 2 ; 16 . Definition generation is further classified into two subcategories, illustrated in Table 1 : (1) contextualized definition generation (CDG), the task of generating definitions of target words given their context, and (2) dictionary definition generation (DDG), the task of generating definitions for all senses of given (head)words. While the present paper focuses on the latter, we review studies on both. Contextualized definition generation Several researchers have applied definition generation techniques to describe unfamiliar words or phrases in context, aiming to support reading specialized documents ( 29 ; 17 ; 14 ; 16 ) or text in a foreign language 42 . An advantage of CDG is its ability to generate definitions well adapted to the context 3 . Dictionary definition generation DDG requires listing all senses of a given word to enable readers to grasp the full picture of its semantics; for inputs, a part of speech and reading may also be given to specify definitions to generate. Resources generated by DDG can also serve as lexicons for various tasks, e.g., word sense disambiguation. Studies on DDG have been conducted in lexicography and computational linguistics. In lexicography, the recent emergence of ChatGPT spurred a wave of interest in computational approaches 7 . 24 explored the use of ChatGPT to create English monolingual dictionary entries containing sense definitions and example sentences. Their evaluation by human experts on 15 entries indicated that the generated definitions are comparable to those created by human lexicographers. 18 generated 99 English dictionary entries using ChatGPT and observed that ChatGPT can write correct and accessible definitions, but it sometimes repeats definitions representing virtually the same sense in different expressions, which is a phenomenon they refer to as “false polysemy.” 35 found that ChatGPT generally outperforms the Oxford Advanced Learner’s Dictionary (OALD) in providing lexicographical data to English language learners. DDG studies in computational linguistics were spearheaded by 19 , who demonstrated the utility of pre-trained multi-sense embeddings in DDG. 28 presented a shared task on DDG on five European languages, with results suggesting that methods based on multi-task learning are effective. 15 investigated Japanese definition generation; however, their study was based on a dataset constructed from Wikipedia and Wikidata, meaning that most of their definitions were descriptions of proper nouns rather than lexicographic definitions of general vocabulary. DDG includes LDDG as a subcategory. Prior studies such as 24 have worked on generating definitions for LDs. However, to our knowledge, no studies have explicitly addressed LDDG as DDG with a vocabulary constraint, as we do. 2.2 Evaluation of DDG The lack of a trustworthy automatic evaluation approach has been a limiting factor in previous work