Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
Craig W. Schmidt, Varshini Reddy & Chris Tanner, Yuval Pinter · 2025
Read Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier by Craig W. Schmidt, Varshini Reddy & Chris Tanner, Yuval Pinter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00178.txt.
Description
Craig Schmidt, Varshini Reddy, Chris Tanner, and Yuval Pinter present BoundlessBPE in two thousand twenty-five. Byte Pair Encoding is a core way to split text into tokens. A first step, pre-tokenization, cuts text on spaces and marks. That step skews token counts toward common full words.
Skew blunts the gain from larger vocabularies in the model. BoundlessBPE relaxes the boundary rule that causes the skew. It merges whole pretokens into units the team calls superwords. Counts across the corpus turn far more even as a result. Text also packs tighter, with more bytes per token.
Catalog record
| CREATOR | Craig W. Schmidt, Varshini Reddy & Chris Tanner, Yuval Pinter |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-00178