Wiki Dumps to Training Corpora: South Slavic Case

Mihailo Škorić [0000-0003-4811-8692] · 2026

Read Wiki Dumps to Training Corpora: South Slavic Case by Mihailo Škorić [0000-0003-4811-8692] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25384.txt.

Description

Škorić and Palma put forth a pipeline that turns Wikimedia dumps into text sets for seven South Slavic tongues. Phase one pulls and cleans text from Wikipedia, Wikisource, Wikibooks, Wikinews, and Wikiquote dumps. Phase two takes on iffy articles, oft made from databases, marked by repeat patterns and stock phrasing. A filter based on n-grams drops highly redundant articles outright. The end sets give rich texts for language models across South Slavic tongues. The approach mostly works for any tongue.

Catalog record

CREATORMihailo Škorić [0000-0003-4811-8692]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39458
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25384

Browse the full library · About the Tower of Babel