Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset
Simon Phetole Ramalepe, Thipe I. Modipa, Marelie H. Davel · 2025
Read Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset by Simon Phetole Ramalepe, Thipe I. Modipa, Marelie H. Davel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15281.txt.
Description
Simon Ramalepe, Thipe Modipa, and Marelie Davel train a text-generating model for Sepedi, a South African language with little digital text, in a research paper from 2025. Low-resource languages are held back by a lack of training data. The team built two new data sets to help: a general Sepedi collection and one from radio news.
They pre-trained transformer models in two ways: with parts of the text hidden, and without hiding. The plain models scored better on loss and perplexity, two standard quality gauges. But when judged on the text they actually produced, the hidden-text models scored slightly higher. News data were used to fine-tune both kinds.
Catalog record
| CREATOR | Simon Phetole Ramalepe, Thipe I. Modipa, Marelie H. Davel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-15281