MegaWika 2: A More Comprehensive Multilingual Collection of Articles and their Sources
Samuel Barham, Chandler May, Benjamin Van Durme · 2025
Read MegaWika 2: A More Comprehensive Multilingual Collection of Articles and their Sources by Samuel Barham, Chandler May, Benjamin Van Durme (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03828.txt.
Introduction
Samuel Barham, Chandler May, and Benjamin Van Durme introduced MegaWika 2 in 2025, a large data set of Wikipedia articles in many languages along with their sources. Each article is stored in a rich structure, and the text of its cited web sources is kept inline, with exact marks showing where each citation sits in the article.
The new version covers six times as many articles as the first and twice as many fully gathered citations. The first version served question answering and search. The new one is aimed at fact checking and at studies of how articles and sources change across time and language. It also still supports report generation research.
Catalog record
| CREATOR | Samuel Barham, Chandler May, Benjamin Van Durme |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 93 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03828
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MegaWika 2: A More Comprehensive Multilingual Collection of Articles and their Sources Samuel Barham, Chandler May, Benjamin Van Durme Abstract Résumé We introduce MegaWika 2 , a large, multilingual dataset of Wikipedia articles with their citations and scraped web sources ; articles are represented in a rich data structure, and scraped source texts are stored inline with precise character offsets of their citations in the article text. MegaWika 2 is a major upgrade from the original MegaWika, spanning six times as many articles and twice as many fully scraped citations. Both MegaWika and MegaWika 2 support report generation research ; whereas MegaWika also focused on supporting question answering and retrieval applications, MegaWika 2 is designed to support fact checking and analyses across time and language. 1 Introduction Collaborative AI for report generation and other complex tasks is in high demand. Wikipedia is an excellent resource for these tasks ; it contains millions of open-access community-authored reports (articles) with claims backed up by external sources (references). Wikipedia articles also contain structured data such as tabular and relational data provided in tables and infoboxes. To our knowledge, existing large-scale Wikipedia datasets do not simultaneously capture rich article structure and fine-grained links between sources and individual sentences or phrases. A large dataset with those features would enable research on a much broader range of problems in report generation, fact checking, and other tasks. Meanwhile, as in-context learning becomes ubiquitous across domains and modalities, access to high-quality, up-to-date corpora for pretraining and tuning large language models (LLMs) becomes vital. Wikipedia is often a backbone for such data, as it contains billions of tokens spanning hundreds of languages. Extracting and scraping sources, linking them to specific claims, and capturing rich article structure would provide more data supporting a larger variety of training and tuning tasks. The original MegaWika dataset consists of 13 million Wikipedia articles spanning 71 million passage-source pairs—30 million with scraped source text—across 50 languages. MegaWika broke ground as a large-scale and highly multilingual report-source dataset Barham et al. 2023 . We introduce MegaWika 2 , a collection of 77 million articles spanning 172 million citations—63 million with scraped source text—across those same 50 languages. MegaWika 2 improves on MegaWika ( MegaWika 1 ) by providing : — more articles, citations, and text ; — sentence-segmented paragraphs ; — web and non-web citations (books, etc.) with text offsets ; — source text quality estimates ; — article elements like tables, infoboxes, and math/code blocks extracted ; — improved translations using NLLB ; and — enrichments like cross-lingual article links and first/last revision dates. We release MegaWika 2 with the aim of accelerating multilingual report generation and related research while facilitating LLM training and tuning. 1 1 1 jhu-clsp/megawika-2 Figure 1: Number of articles (log scale) per language in MegaWika 1 and MegaWika 2. Languages are represented by their two-letter Wikipedia project codes (equivalent to ISO 639-1 language codes for these 50 languages) and are ordered by their number of articles in MegaWika 1. 2 What’s Changed MegaWika 2 improves on MegaWika 1 in data quality and quantity. MegaWika 2 captures rich article structure, representing citations in the context of the article they appear in ; it includes text quality estimates for articles’ sources ; it provides improved English translations of article text ; it contains many more articles, citations, and scraped sources ; and it adds data enrichments. Improved Article Representation MegaWika 1 focused primarily on the relationship between individual article passages and the web sources they cite, which it represented as a list of passage-source pairs (one source per passage). MegaWika 1 also provided the full text for each article, but the passages and citations were not represented explicitly in that broader context. By contrast, MegaWika 2 captures the structure of the article, detecting paragraph boundaries and exposing headings, paragraphs, tables, infoboxes, and more. Each paragraph contains a list of sentences, each of which contains a list of citations and citation-needed markers with corresponding character offsets. Sources are thus represented in the context of the surrounding article and their precise locations in the text are preserved. Moreover, all citations are captured, not just citations containing URLs (however, only web citations’ content is scraped and stored in the data set). The MegaWika 2 dataset is described in further detail in Section 4 . Source Text Quality Estimates Extracted source text in MegaWika 1 was often low-quality, making the sources difficult to use in an automated fashion in downstream research. Many of the circumstances underlying these low-quality sources are effectively unavoidable, including paywalls and CAPTCHAs, other access-restricted pages, and deleted or renamed pages. To mitigate this issue in MegaWika 2, we provide five-level ordinal text quality estimates for all scraped sources. Our text quality estimates are tuned for precision, enabling users to reliably filter out lower-quality source text as desired. Improved Translation MegaWika 1 used M2M100 to translate citation-terminated passages into English, using Google Translate for lower-resource languages where M2M100’s performance was unsatisfactory. MegaWika 2 uses NLLB-200 (600 million parameters) to translate all sentences and headings to English, providing improved translation performance overall, as discussed in Section 4.4 . Additionally, MegaWika 2 contains over five times as many sentences (and headings) as MegaWika 1, so the workload is now much larger. Accessing a pool of 16 shared NVIDIA H100 GPUs on the Johns Hopkins University Data Science and AI Institute (DSAI) grid, we were able to translate the one billion sentences and headings of MegaWika 2 in just over two weeks, whereas translating the 180 million sentences in MegaWika 1 took months. Increased Quantity MegaWika 2 has six times as many articles, over five times as much text—one billion sentences and headings compared to 180 million sentences—and twice as many citations with text extracted as MegaWika 1. See Figure 1 for a breakdown of the number of articles in each data set by language, and see Section 4.2 for a detailed analysis of citations. 3 Data Processing MegaWika 2 data processing is divided into four pipelines, allowing us to run data processing steps with different hardware requirements on different systems. 1. First, the extraction pipeline downloads a Wikipedia dump, extracts the articles, organizes the collection into 1000 -article chunk files, parses the Wikitext code (Wikipedia markup code) of each article, identifies citations, and, for citations that point to a URL, downloads the HTML code or plain text at that URL and extracts text content from it. This pipeline is CPU- and memory-intensive. 2. Next, the translation pipeline extracts headings and sentences from articles in a chunk, translates them to English, and reinserts them into the collection. This pipeline is GPU-intensive. 3. The source quality pipeline extracts source texts, estimates source text quality, and inserts the estimates back into the collection. It is also GPU-intensive. 4. Finally, the enrichment pipeline extracts editor-provided quotes from citations, fetches cross-lingual links between each article and parallel articles in other Wikipedia projects (languages) as well as the creation date of each article. This pipeline’s bottleneck is the Wikipedia Action API rate limit. We implement these pipelines using Luigi , a Python framework that facilitates designing and running complex data pipelines on a single machine (with tools for invoking external