Slovak morphological tokenizer using the Byte-Pair Encoding algorithm
Dávid Držík, Frantisek Forgac · 2024
Read Slovak morphological tokenizer using the Byte-Pair Encoding algorithm by Dávid Držík, Frantisek Forgac (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-11622830.txt.
Description
David Drzik and Frantisek Forgac build a tokenizer for the Slovak language that respects word roots, in a 2024 study. Tokenizers chop text into pieces for language models, and the standard method can slice through a root, blurring a word’s core meaning. Slovak has rich word endings, so the risk is high.
Their method first pulls roots from morphology dictionaries, then trains the standard chopping algorithm around them. In tests it kept roots intact in 99.7 percent of cases, against about 90.5 percent for a Slovak BERT tokenizer and 93.1 percent for a plain baseline. Models fine-tuned with it also scored better on a sentiment sorting task. Cleaner pieces should help any Slovak language tool.
Catalog record
| CREATOR | Dávid Držík, Frantisek Forgac |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |