Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier

Craig W. Schmidt, Varshini Reddy & Chris Tanner, Yuval Pinter · 2025

Read Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier by Craig W. Schmidt, Varshini Reddy & Chris Tanner, Yuval Pinter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00178.txt.

Description

Craig Schmidt, Varshini Reddy, Chris Tanner, and Yuval Pinter present BoundlessBPE in two thousand twenty-five. Byte Pair Encoding is a core way to split text into tokens. A first step, pre-tokenization, cuts text on spaces and marks. That step skews token counts toward common full words.

Skew blunts the gain from larger vocabularies in the model. BoundlessBPE relaxes the boundary rule that causes the skew. It merges whole pretokens into units the team calls superwords. Counts across the corpus turn far more even as a result. Text also packs tighter, with more bytes per token.

Catalog record

CREATORCraig W. Schmidt, Varshini Reddy & Chris Tanner, Yuval Pinter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00178

Browse the full library · About the Tower of Babel