Wikipedia is Not a Dictionary, Delete! Text Classification as a Proxy for Analysing Wiki Deletion Discussions
Hsuvas Borkakoty, Luis Espinosa-Anke · 2025
Read Wikipedia is Not a Dictionary, Delete! Text Classification as a Proxy for Analysing Wiki Deletion Discussions by Hsuvas Borkakoty, Luis Espinosa-Anke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10294.txt.
Introduction
When an article on Wikipedia is marked for deletion, editors argue about its fate in long public threads. Hsuvas Borkakoty and Luis Espinosa-Anke turned those threads into data. They built a bank of such talks from several wikis in three tongues. Language models then tried a set of tasks on the talks. The models had to guess how each talk would end. They also had to guess which rule each remark was pointing to, in a quiet way.
Talks that ended in deletion were the easiest to call. Oddly, the tags editors stamp on their own remarks, keep, delete, or redirect, did not always help the models. People waver as they argue, and their words can fight their tags. The study uses these tests as a lens on how the crowd governs itself, and on how machines might one day help with the task.
Catalog record
| CREATOR | Hsuvas Borkakoty, Luis Espinosa-Anke |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10294
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Wikipedia is Not a Dictionary, Delete! Text Classification as a Proxy for Analysing Wiki Deletion Discussions Hsuvas Borkakoty, Luis Espinosa-Anke Abstract Automated content moderation for collaborative knowledge hubs like Wikipedia or Wikidata is an important yet challenging task due to multiple factors. In this paper, we construct a database of discussions happening around articles marked for deletion in several Wikis and in three languages, which we then use to evaluate a range of LMs on different tasks (from predicting the outcome of the discussion to identifying the implicit policy an individual comment might be pointing to). Our results reveal, among others, that discussions leading to deletion are easier to predict, and that, surprisingly, self-produced tags (keep, delete or redirect) don’t always help guiding the classifiers, presumably because of users’ hesitation or deliberation within comments 1 1 1 Dataset available at: https://huggingface.co/datasets/hsuvaskakoty/wider . . 1 Introduction Wikipedia and its sister Wikis play an indispensable role as a collaborative knowledge source, and are widely used by students Selwyn and Gorard (2016) and the general public Singer et al. (2017) ; Lemmerich et al. (2019) alike. They fulfill use cases that range from core knowledge go-tos , as well as ‘‘free’’ supporting documentation for content providers and search engines (e.g., Google 2 2 2 https://en.wikipedia.org/wiki/Relationship_between_Google_and_Wikipedia or YouTube 3 3 3 https://support.google.com/youtube/answer/7630512?hl=en ). However, due to their size and, most importantly, their collaborative nature, ensuring high quality in these platforms is challenging, especially given the need to “map” content to existing policies at least semi-automatically Ribeiro et al. (2022) . This is particularly relevant in the GenAI era, as AI-generated content has proliferated throughout the Internet Brooks et al. (2024) . Figure 1: Example of a Deletion Discussion in English Wikipedia Task Lang. Platform Label Set Outcome Prediction En Wikipedia delete, keep, redirect, no-consensus, merge, speedy keep, speedy delete, withdrawn Wikidata-ent. delete, keep, merge, redirect, no-consensus, comment Wikidata-pr. delete, keep, no-consensus Wikiquote delete, keep, redirect, merge, no-consensus Wikinews delete, keep, speedy delete, comment Es Wikipedia borrar (delete), mantener (keep), fusionar (merge), otros (others) Gr Wikipedia \acctonos (delete), \acctonos ς (keep), \acctonos ς \acctonos ς (no-consensus) Stance Detection En Wikipedia delete, keep, merge, comment Policy Prediction En Wikipedia Wikipedia:Notability, Wikipedia:What Wikipedia is not, Wikipedia:No original research, Wikipedia:Verifiability, Wikipedia:Arguments to avoid in deletion discussions, Wikipedia:Biographies of living persons, Wikipedia:Criteria for speedy deletion, Wikipedia:Articles for deletion, Wikipedia:Wikipedia is not a dictionary, Wikipedia:Deletion policy Table 1: Label set for the different tasks and datasets we consider in this paper (Outcome, Stance, and Policy), for three languages (English: En, Spanish: Es, and Greek: Gr) and five (4+1) platforms (Wikipedia, Wikidata-Entity (ent.) and Property (pr.), Wikinews, and Wikiquote). More generally, content moderation in online platforms is often the outcome of group coordination and communication Chidambaram and Tung (2005) ; Jensen and Scacchi (2005) ; Butler et al. (2008) . Unsurprisingly, NLP plays an important role in automating this process. For example, Singhal et al. (2023) defined a framework for social media moderation as a function of community guidelines, policy enforcement and violation detection. Some prominent examples of works connected with such a framework are: policy based content moderation in Facebook Sablosky (2021) , rule-breaking behavior analysis on Reddit Chandrasekharan et al. (2018) , and topic based content moderation discourse on X Alizadeh et al. (2022) . In the case of Wikis, both the guidelines and the rules that govern the quality of their content are maintained by contributions from the community Seering (2020) , where discussion-based approaches towards content moderation are the norm. The way this generally works is that, given an article flagged by a community member, users justify their stance towards it and its adherence to the policies, and then editors act. However, manual efforts to clear the backlog are insufficient. Therefore, NLP techniques for predicting the outcome of a deletion discussion, or for capturing a user’s stance towards a specific article are critical Mayfield and Black (2019b) ; Kaffee et al. (2023) . Despite this need, there is a surprising lack of work beyond Wikipedia, and there is almost no published work that looks at non-English languages (with the exception of Kaffee et al. (2023) ). Moreover, in-depth comparative analyses of parameter-efficient techniques have also been so far largely unexplored. We therefore aim to address all of the above with an analysis on a novel collection of deletion discussions, in three languages and four platforms. These discussions come, if resolved, alongside the discussion outcomes (generally speaking, suggesting to keep or delete the article, although the actual outcome tags are more fine grained than this), with individual comments having their own stance and referring to specific policies Kaffee et al. (2023) . Our classification experiments set strong baseline results for the community to build upon, and provide insights into these community-led activities. Language Platform Total en Wikipedia 18,528 Wikidata-entities 355,428 Wikidata-properties 498 Wikinews 91 Wikiquote 695 es Wikipedia 3,274 gr Wikipedia 392 Table 2: Overall number of deletion discussions per language and platform in the outcome prediction dataset. Platform Language Example title Discussion (truncated) Outcome Wikipedia en Beast Poetry Editor 1: Keep in one form or another. Editor 2: One option could be to re-frame the article to be about the book. Keep Wikidata-Ent en Q28090948 Editor 1: no description : Vandalism. Delete Wikidata-Prop en JMdict sequence number (P11700) Editor 1: Deleted - ( ) Support I assume no comments have been made because this is a clear case to delete… Delete Wikinews en Mugalkhod Jeedga Mutta organizes mass marriage in Belgaum, India Editor 1: There is no further meaningful work on the article. Editor 2: All advice ignored, kill it with cleansing fire and stop wasting time. Speedy delete Wikiquote en 3rd Rock From The Sun Editor 1: Two reasons to delete this: - It is a copy of about half of quotes on IMDB - 3rd Rock from the Sun is a different article. Editor 2: Merge, perhaps with some trimming. Merge Wikipedia-Es es Héroes: El legado de la Evolución Editor 1: Bórrese Irrelevante enciclopédico. Editor 2: Bórrese Irrelevante. Borrar Wikipedia-gr gr ς \acctonos \acctonos ςς \acctonos Editor 1: \acctonos \acctonos \acctonos µ \acctonos / \acctonos ς « \acctonos \acctonos \acctonos \acctonos \acctonos \acctonos µς… Editor 2: µ \acctonos \acctonos \acctonos µ \acctonos \acctonos \acctonos ς µ \acctonos . \acctonos Table 3: Examples for different platforms and languages, alongside outcome labels. 2 Tasks and dataset Construction We consider three tasks, namely (1) Outcome prediction - given a full discussion around an article marked for deletion, predict the final decision; (2) Stance detection , i.e. given an individual comment, determine its stance towards the decision to be made for that article; and (3) Policy prediction , where again, given one single comment, we want to determine the policy that comment is most likely be referring to (Figure 1 shows an exmaple). We build a novel dataset for outcome prediction, while for the other two tasks we largely rely on the dataset from Kaffee et al. (2023) (although with some important modifications to enable the goal of this paper, namely an in-depth analysis). We provide more detail about these datasets in the following subsections. 2.1 Outcome Prediction We retrieve and clean deletion discussions programmatically 4 4 4 We