Adapt BPE : From General Purpose to Specialized Tokenizers
Vijini Liyanage, François Yvon · 2026
Read Adapt BPE : From General Purpose to Specialized Tokenizers by Vijini Liyanage, François Yvon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21665.txt.
Description
Vijini Liyanage and François Yvon adapt word-piece tokenizers to special domains. Tokenization chops text into pieces that a language model reads. Standard practice trains one general tokenizer for everything, then uses it for training and serving alike.
A generic piece set wastes effort on text from narrow fields or specific languages. Rare terms get shredded into many tiny pieces, which costs speed and accuracy. The authors adapt the popular BPE method to build specialized tokenizers. A fitted tokenizer could make domain models, from law to medicine, cheaper to run and sharper on their own vocabulary.
Catalog record
| CREATOR | Vijini Liyanage, François Yvon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21665