Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark
2026
Read Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12744.txt.
Introduction
Terra Blevins and 13 colleagues build Universal NER v2, a multilingual named entity benchmark. Gold-standard test data remain scarce in most languages. The Universal NER project, now in its fourth year, collects standard cross-lingual annotations. A general tagset and thorough guidelines govern named entity spans. UNER v1, the first installment, appeared in 2024. The project has since grown into an active community.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26913 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.12744
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark Abstract We present Universal NER (UNER) v2, a significant extension of the initial version released in 2024. UNER is a collaborative dataset for multilingual named-entity annotations, built to support research on NER methods in a cross-linguistic setting. UNER v2 adds 11 new datasets in 10 typologically varied language varieties to the resource, including multiple parallel evaluation benchmarks aligned with each other and other datasets in UNER v1, while maintaining the same annotation guidelines and high standards for inter-annotator agreement. We report detailed statistics for the dataset and benchmark UNER v2 using both encoder-based model architectures and LLMs. Keywords: Named Entity Recognition, Multilingual, Benchmark, Cross-lingual Transfer, Large Language Models Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark Terra Blevins 1 , Stephen Mayhew 2 , Marek Šuppa 3 , Hila Gonen 4 , Shachar Mirkin 5 , Vasile Pais 6 , Kaja Dobrovoljc 7 , Voula Giouli 8 , Jun Kevin 9 , Eugene Jang 1 , Eungseo Kim 10 , Jeongyeon Seo 11 , Xenophon Gialis 12 , Yuval Pinter 13 1 Northeastern University, USA 2 Duolingo, USA 3 Comenius University Bratislava, Slovakia 4 University of British Columbia, Canada 5 Alpinference, France 6 Research Institute for Artificial Intelligence, Romanian Academy, Romania 7 University of Ljubljana, Slovenia 8 Aristotle University of Thessaloniki / ILSP, Athena Research Center, Greece 9 Universitas Pelita Harapan, Indonesia 10 Seoul National University, South Korea 11 Independent Researcher, South Korea 12 Democritus University of Thrace, Greece 13 Ben-Gurion University of the Negev, Israel [email protected], [email protected], [email protected] , [email protected], [email protected] [email protected], [email protected], [email protected] [email protected], [email protected] [email protected], [email protected], [email protected], [email protected] Abstract content 1. Introduction While multilingual language models promise to bring the benefits of LLMs to speakers of many languages, gold-standard evaluation benchmarks in most languages to interrogate these assumptions remain scarce. The Universal NER project, now entering its fourth year, is dedicated to building gold-standard multilingual Named Entity Recognition (NER) benchmark datasets. Inspired by existing massively multilingual efforts for other core NLP tasks ( 11 , e.g., Universal Dependencies;) , the project uses a general tagset and thorough annotation guidelines to collect standardized, cross-lingual annotations of named entity spans. The first installment (UNER v1) was released in 2024 20 , and the project has continued and expanded since then, with various organizers, annotators, and collaborators in an active community. Figure 1: Comparison of the dataset statistics for UNER v1 and UNER v2. Language Dataset el Greek gdt Greek Dependency Treebank he Hebrew htb Hebrew Dependency Treebank nno Norwegian Nynorsk ndt Norwegian Dependency Treebank nob Norwegian Bokmål ndt Norwegian Dependency Treebank sl Slovenian ssj Slovenian SSJ Treebank sv Swedish lines Swedish LinES Treebank cs Czech pud Parallel Universal Dependencies id Indonesian pud Parallel Universal Dependencies ja Japanese pud Parallel Universal Dependencies ko Korean pud Parallel Universal Dependencies ro Romanian legalnero Romanian Named Entities in the Legal Domain Table 1: Languages and associated original dataset names used in UNER v2. We present a substantial update to the Universal Named Entity Recognition (UNER) project, with new, gold-standard annotations on eleven datasets in ten languages (nine of which are new to the UNER collection), comprising 59,000 entities over 1.5 million tokens. As shown in Figure 1 , the addition of these datasets brings the composition of the overall Universal NER repository to 30 datasets across 22 languages, with a total of 3 million annotated tokens. We release this aggregated dataset as UNER v2. In this work, we first provide a summary of the Universal NER project thus far (Section 2 ) and then describe the creation of UNER v2 datasets, along with relevant dataset statistics and analyses (Section 3 ). We then present baseline experiments on the new datasets introduced in UNER v2, benchmarking them against both encoder-based and generative language models (Section 4 ). Our experimental results show that state-of-the-art generative models achieve at most 0.50 average F1, with performance dropping sharply on typologically distant and lower-resourced languages. UNER continues to be an essential benchmark for developing more robust, truly multilingual NER systems. 2. Universal NER The development of multilingual Named Entity Recognition benchmarks has accelerated in recent years. The Universal NER project ( 20 , UNER v1) established a community-driven gold standard resource in 13 languages, following the philosophy of projects such as Universal Dependencies ( 11 , UD;) or the PARSEME initiative for multiword expressions 26 . Complementary efforts have also emerged in domain-specific contexts, such as LegalNERo 24 , which focuses on the Romanian legal domain, and NorNE 16 , which contains named entity annotations for Norwegian. The hallmark of the Universal NER project is a simple and transferable tagset consisting of 3 tags – Person ( per ), Location ( loc ), and Organization ( org ) – as well as thorough annotation guidelines. Annotation is performed using shared software based on TALEN 21 , and gathered on GitHub. To encourage high-quality annotations, we require (with exceptions) that at least 2 annotators provide annotations on at least 5% of the data, so that we can calculate agreement statistics. In many cases, we have found that even when there is low overlap between annotators, agreement statistics can uncover annotation guideline misunderstandings or incompetent annotators, and identify potential quality issues. Impact of UNER v1 Since its introduction in 2024, the first version of this dataset has had a substantial impact on the multilingual NLP community, garnering citations from over 40 research works. Its significance is further evidenced by its integration into major multilingual resources: the Aya Dataset, a prominent open-access collection for multilingual instruction tuning 27 , and skLEP, a comprehensive Slovak language understanding benchmark 30 . Other works have leveraged the first version of the dataset to investigate targeted questions—such as the effect of NER on translation 28 and the efficacy of synthetic data for NER tasks 17 —as well as broader fundamental questions concerning cross-lingual transfer 19 ; 8 and multilingual knowledge distillation 37 . The dataset continues to enable explorations across a wide spectrum of multilingual research directions 1 ; 13 ; 18 ; 29 ; 4 . Sentences Entities Tokens Lang. Dataset Train Dev Test All Train Dev Test All Train Dev Test All el gdt 1,662 403 456 2,521 1,551 501 436 2,488 42,326 10,443 10,672 63,441 he htb 5,241 484 491 6,216 6,013 434 439 6,886 137,717 11,412 12,282 161,411 nno ndt 14,174 1,890 1,511 17,575 10,348 1,111 897 12,356 245,330 31,250 24,773 301,353 nob ndt 15,696 2,409 1,939 20,044 10,062 1,438 1,259 12,759 243,886 36,369 29,966 310,221 sl ssj 10,903 1,250 1,282 13,435 7,293 907 724 8,924 215,155 26,500 25,442 267,097 sv lines 3,176 1,032 1,035 5,243 1,221 433 451 2,105 55,451 18,515 16,994 90,960 cs pud – – 1,000 1,000 – – 1,008 1,008 – – 18,610 18,610 id pud – – 1,000 1,000 – – 1,110 1,110 – – 19,446 19,446 ja pud – – 1,000 1,000 – – 1,150 1,150 – – 28,788 28,788 ko pud – – 1,000 1,000 – – 1,172 1,172 – – 16,584 16,584 ro legalnero – – 8,284 8,284 – – 8,996 8,996 – – 265,335 265,335 Table 2: Dataset statistics for the new benchmarks included in UNER v2. 3.