Multilingual Reference Need Assessment System for Wikipedia Conference: ; ; CCS: Computing methodologies Machine learning CCS: Computing methodologies Natural language processing CCS: Information systems Wikis CCS: Information systems Language models

Aitolkyn Baigutanova, Francisco Navas, Pablo Aragón, Mykola Trokhymovych, Muniza Aslam, Ai-Jou Chou, Miriam Redi, Diego Sáez Trumper · 2026

Read Multilingual Reference Need Assessment System for Wikipedia Conference: ; ; CCS: Computing methodologies Machine learning CCS: Computing methodologies Natural language processing CCS: Information systems Wikis CCS: Information systems Language models by Aitolkyn Baigutanova, Francisco Navas, Pablo Aragón, Mykola Trokhymovych, Muniza Aslam, Ai-Jou Chou, Miriam Redi, Diego Sáez Trumper (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17146.txt.

Introduction

Baigutanova and co-authors build a multilingual system that flags Wikipedia claims in need of references. Wikipedia's verifiability rule asks that claims be backed by reliable sources, but editors must check this by hand amid a flood of daily edits. Their machine learning system assists editors by spotting claims that require citations. It was tested in 10 language editions and beat existing benchmarks for reference need assessment. The team weighs accuracy against computing cost under real infrastructure limits. The system is deployed in production, with data and code released.

Catalog record

CREATORAitolkyn Baigutanova, Francisco Navas, Pablo Aragón, Mykola Trokhymovych, Muniza Aslam, Ai-Jou Chou, Miriam Redi, Diego Sáez Trumper
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36394
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.17146

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multilingual Reference Need Assessment System for Wikipedia Conference: ; ; CCS: Computing methodologies Machine learning CCS: Computing methodologies Natural language processing CCS: Information systems Wikis CCS: Information systems Language models Aitolkyn Baigutanova, Francisco Navas, Pablo Aragón, Mykola Trokhymovych, Muniza Aslam, Ai-Jou Chou, Miriam Redi, Diego Sáez Trumper Abstract . Wikipedia is a critical source of information for millions of users across the Web. It serves as a key resource for large language models, search engines, question-answering systems, and other Web-based applications. In Wikipedia, content needs to be verifiable, meaning that readers can check that claims are backed by references to reliable sources. This depends on manual verification by editors, an effective but labor-intensive process, especially given the high volume of daily edits. To address this challenge, we introduce a multilingual machine learning system to assist editors in identifying claims requiring citations. Our approach is tested in 10 language editions of Wikipedia, outperforming existing benchmarks for reference need assessment. We not only consider machine learning evaluation metrics but also system requirements, allowing us to explore the trade-offs between model accuracy and computational efficiency under real-world infrastructure constraints. We deploy our system in production and release data and code to support further research. 1. Introduction Verifiability is one of Wikipedia’s core content policies ( 32 ) , ensuring that information can be checked against reliable sources. In practice, this policy is frequently unmet, resulting in numerous claims without citations. As of September 2025, more than 550k articles on English Wikipedia contain one or more statements marked with a {{ Citation needed }} tag ( 30 ) . To address this issue, Wikipedia editors employ the {{ Citation needed }} tag ( 31 ) , which signals users that a statement requires a supporting source. Given the high volume of edits—approximately six per second as of 2024 ( 33 ) —manually identifying and tagging uncited claims is prone to delays. Computational support is therefore essential to assist Wikipedia editors in maintaining verifiability standards. As pointed out by the Wikimedia Foundation’s recently published AI Strategy ( 10 ) , open machine learning models that aid editors in preserving knowledge integrity are crucial to maintain Wikipedia’s quality. Although claim detection on Wikipedia has been studied before, most prior research has focused on the English edition ( 22 ; 36 ) , with only a few case studies on smaller and less-resourced language editions. With Wikipedia spanning over 300 languages, there remains a need to tackle this problem in a multilingual context ( 17 ) . Furthermore, automated detection of unsupported claims on Wikipedia, which covers a broad range of topics, has not yet been explored in production-level settings. Given the need to adjust accuracy, latency, and resource availability, addressing the practical challenges of implementing such systems in real-world environments is crucial. Figure 1 . The Reference Need pipeline to compute the reference coverage score of an article revision. In this work, we adopt the previously proposed reference need definition ( 3 ) for a real-world product use case and introduce a novel machine learning system specifically designed to assess the reference coverage of Wikipedia articles at scale. The model is deployed with an initial use case aimed at assisting external re-users in evaluating the verifiability of Wikipedia articles. We further provide industrial insights into language modeling strategies by deploying the system under strict resource constraints, with particular focus on balancing model accuracy and system performance. In summary, our contributions are as follows: • We introduce an open-source multilingual system for evaluating the reference need of Wikipedia articles, evaluated on the top 10 language editions by active user count. • We show the trade-offs between accuracy and latency when deploying language models in production under strict resource constraints. In this context, we explore model optimizations and compare small language models (SLMs) with large language models (LLMs) for this task, showing that SLMs may offer an optimal compromise between precision and serving time. • We release the training 1 1 1 https://gitlab.wikimedia.org/repos/research/reference-quality/-/tree/research-notebooks/RN and inference code 2 2 2 https://gitlab.wikimedia.org/repos/research/reference-quality/-/tree/classifiers/reference_quality/models/ref_need , along with the data 3 3 3 https://analytics.wikimedia.org/published/datasets/one-off/reference-need used in our experiments, enabling further research in the field of information credibility on the Web. 2. Related Work Our system is designed to improve existing models in identifying reference needs on Wikipedia, leveraging recent advancements in language models. 2.1. References on Wikipedia The body of research work analyzing Wikipedia citation practices is growing, including datasets of extracted citations ( 1 ; 18 ) and studies of the source quality ( 4 ; 37 ) . 22 contributed to predicting citation needs on Wikipedia by developing a taxonomy and a deep learning model, later integrated with Wikimedia community tools ( 5 ) . The main limitation of these works is their focus on English, despite growing interest in multilingual research ( 17 ) . This is important for resource-limited language communities, where identifying citation needs is challenging due to scarce ground-truth data. 12 ’s approach on three small Wikipedias has shown progress, but remains untested in production, leaving deployment challenges unaddressed. 2.2. Masked Language Models Language models have become pivotal in production-level natural language processing workflows ( 14 ) . Masked language models (MLMs) like BERT ( 8 ) and RoBERTa ( 19 ) have been successful in tasks such as sentiment classification ( 21 ) , readability assessment ( 27 ) , and question answering ( 38 ) . They scale well in multilingual contexts, efficiently handling over 100 languages with a single model ( 6 ) . However, MLMs require substantial resources for production-scale use. Techniques like knowledge distillation ( 13 ) , quantization ( 11 ) , and pruning ( 39 ) produced more efficient models. DistilBERT reduced model size by 40% while retaining 97% of language understanding, boosting speed by 60% ( 23 ) . We explore the application of these methods to the reference need task. 2.3. Large Language Models While modern LLMs can perform zero- and few-shot classification via simple prompting ( 28 ) , eliminating the need for extensive training, they require substantial computational resources, limiting their accessibility. Open-weight models, such as Llama ( 25 ; 26 ) and Mistral ( 16 ) , were released, but their deployment remains costly. Assessing LLM performance against smaller fine-tuned MLMs is essential to estimate their suitability for our task. Moreover, recent research has shown the limitations of LLMs in applying complex Wikipedia policies, such as assessing content neutrality ( 2 ) , suggesting that this is a complex task that requires domain knowledge. Our work confirms these limitations and presents a comparison between large and small models, showing that smaller models can reach similar performance compared with their larger counterparts, require fewer resources, and offer faster serving time. 3. Reference Need 3 defined Reference Need (RN) of a Wikipedia article as the proportion of uncited claims among those that require a reference. We adapt the original formulation, which considers all sentences in an article, to focus on a set of uncited sentences for prediction. Due to infrastructure constraints described below, we omit cited sentences from our evaluation pipeline because: 1) it notably reduces the number of sentences to evaluate, speeding up the inference process,

Read the complete text