Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset
2026
Read Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00411.txt.
Introduction
Alistair Plum, Laura Bernardy, and Tharindu Ranasinghe present a dataset for named entity recognition in Luxembourgish, a language with few digital resources. Systems that tag names of people, places, and organizations need labeled text, which is costly to make for smaller languages. Their 2026 paper describes judgeWEL, a corpus built mostly by machine, then checked by large language models.
The starting labels come from Wikipedia links, whose types are inferred from linked Wikidata entries. Because those labels are uneven, several language models judge and filter them, keeping only high-quality sentences. The finished corpus is about five times larger than the Luxembourgish dataset available before. It also covers entity types more evenly, giving low-resource language research a stronger base.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-00411
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset Abstract We present judgeWEL, a dataset for named entity recognition (NER) in Luxembourgish, automatically labelled and subsequently verified using large language models (LLM) in a novel pipeline. Building datasets for under-represented languages remains one of the major bottlenecks in natural language processing, where the scarcity of resources and linguistic particularities make large-scale annotation costly and potentially inconsistent. To address these challenges, we propose and evaluate a novel approach that leverages Wikipedia and Wikidata as structured sources of weak supervision. By exploiting internal links within Wikipedia articles, we infer entity types based on their corresponding Wikidata entries, thereby generating initial annotations with minimal human intervention. Because such links are not uniformly reliable, we mitigate noise by employing and comparing several LLMs to identify and retain only high-quality labelled sentences. The resulting corpus is approximately five times larger than the currently available Luxembourgish NER dataset and offers broader and more balanced coverage across entity categories, providing a substantial new resource for multilingual and low-resource NER research. Keywords: Named Entity Recognition, Under-Represented Languages, Large Language Model Evaluation Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset Alistair Plum ♢ , Laura Bernardy ♢ , Tharindu Ranasinghe ♠ ♢ University of Luxembourg, Luxembourg ♠ Lancaster University, UK {alistair.plum, laura.bernardy}@uni.lu [email protected] Abstract content 1. Introduction Recent advances in large language models (LLMs) have redefined the scope of natural language processing (NLP), shifting attention from classification and sequence-labelling tasks to increasingly complex generative applications. This transformation has enabled conversational systems that integrate reasoning, translation, and summarisation within a single multi-task architecture 7 ; 52 ; 19 . Yet, the reach of these technologies remains uneven. For many less-resourced and under-represented languages, the benefits are limited: they continue to lack fundamental resources such as labelled corpora that support even basic supervised learning tasks, including sequence labelling and text classification 20 ; 2 . Building datasets for such languages remains particularly challenging. Obstacles typically include a scarcity of written material, limited linguistic expertise, and the absence of sustained funding, all of which reinforce a cycle of under-representation. When coupled with limited prior research, these constraints make it difficult to bootstrap the resources necessary for progress 17 . In these circumstances, creative and data-efficient strategies for dataset construction are essential. This is the case for Luxembourgish, the national language of Luxembourg and one of three official languages alongside German and French. Despite its official status, Luxembourgish is the only national language of an EU member state not represented at the EU administrative level, thus remaining absent from standardised resources such as Europarl 23 . Luxembourgish is under-represented in NLP 20 compared to its larger neighbours German and French, though it shares deep historical and structural ties with both. It evolved from the Moselle Franconian dialect 16 , displaying strong affinities with German while integrating extensive French lexical and grammatical. Code-switching and borrowing are common, particularly in informal written communication, giving the language a dynamic character that further complicates computational treatment. Despite these challenges, the language’s roughly 400,000 speakers 16 , ongoing standardisation efforts, and renewed digital interest make this an opportune moment to improve NLP for Luxembourgish. In this paper, we present a new methodology for the automatic creation of a named entity recognition (NER) dataset. The methodology is designed to achieve broad and reliable coverage with minimal human supervision. To overcome the bottleneck of manual annotation, we combine distant supervision from Wikipedia 59 and Wikidata 56 with LLM-based quality control, leveraging the ability of current large models to judge annotation consistency even in languages not officially supported. We hypothesise that certain LLMs can reliably distinguish high-quality annotated sentences from those containing missing or incorrect tags, thus enabling efficient filtering and scalable dataset construction. Our main contributions 1 1 1 The dataset and code will be made publicly available after the review process. are as follows: (1) An open pipeline for automatically constructing annotated NER datasets using Wikipedia, Wikidata, and LLM-based judgements. (2) The judgeWEL dataset for Luxembourgish NER, consisting of 28,866 sentences featuring five entity types. 2. Related Works In this section, we summarise related works in the field of NER. Section 2.1 covers general work, Section 2.2 approaches in low-resource (LR) settings, and Section 2.3 focuses on resources for Luxembourgish. This includes existing language resources, both in terms of data and models. 2.1. Named Entity Recognition NER is a fundamental Information Extraction and NLP task that identifies and classifies entities (e.g., people, locations, organizations, dates) via sequence labeling schemes such as BIO 43 . It builds the base for applications like question answering, translation, summarization, and knowledge base construction. Research on NER was started with MUC-6 50 . Early rule-based systems used linguistic heuristics 32 ; 57 and gazetteers 46 ; 13 ; 33 , the latter forming the basis for knowledge-based methods exploiting ontologies 31 ; 47 . These approaches required no labelled data and training but lacked (cross-domain) scalability. Recent work explores generative and LLM approaches. Zero-shot NER with LLMs as standalone remains challenging 60 ; 18 , but improvements are achieved through prompt engineering and few-shot learning with approaches like PromptNER 4 , GL-NER 64 , and ProML 9 . Hybrid frameworks like SuperICL 61 and LinkNER 28 further integrate LLM reasoning with fine-tuned NER models to enhance accuracy. 2.2. Named Entity Recognition for Low-Resource languages NER systems for high-resource (HR) languages such as English have achieved state-of-the-art performance due to the availability of large annotated datasets, including CoNLL’03 51 and OntoNotes 58 . However, reaching this performance is difficult in LR settings where labelled data is scarce. This lack of annotated data is often mitigated through data augmentation and knowledge transfer between HR and LR languages. Cross-lingual knowledge transfer aims to transfer learned linguistic and semantic representations from a source language to a target language with limited labelled data. For example, 30 used translation techniques based on the lexica and Wikipedia to map the knowledge from HR to LR languages. Similarly, 14 used an extended approach by enriching it with semantic mappings, enabling coverage of named entities unseen in the LR datasets. 54 developed a technique for cross-lingual Wikification, linking non-English Wikipedia entries to English Wikipedia, training multilingual embeddings based on it to facilitate cross-lingual transfer. The WikiAnn dataset 38 is a large multilingual benchmark covering 282 languages, also created via cross-lingual Wikipedia links. Likewise, Polyglot-NER leveraged distributed word representations encoding semantic and syntactic properties of words across languages, by integrating Wikipedia links and Freebase attributes to create a multilingual NER corpus in 40 languages 3 . An intermediate approach between cross-lingual transfer and augmentation is MulDA 26 . It addresses the absence of labelled data in a target language by translating limited NER data into multiple target languages using entity placeholders, then generating synthetic multilingual sequences through LMs. For low-resource NER augmentation, PromptDA 25 introduced a label-conditioned word replacement and question-answering style prompting method using BERT. 8 enhanced LR NER by fine-tuning XLM-RoBERTa using data from HR languages and an attention-based strategy to capture the most relevant semantic and syntactic informationMore recent augmentation methods employ LLMs for contextual and entity-level rewriting 62 , designed specifically