Peek2: A Regex-free implementation of pretokenizers for Byte-level BPE
Liu Zai · 2026
Read Peek2: A Regex-free implementation of pretokenizers for Byte-level BPE by Liu Zai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05833.txt.
Introduction
Liu Zai and Iraklis Klampanos introduce Peek2 in 2026, a faster pretokenizer for byte-level byte-pair encoding on edge devices. Pretokenization is the sequential first pass that splits text before token encoding, and it had received little optimization attention for on-device inference. Their replacement targets the pretokenizers used by several major model families.
After analyzing the original logic, the authors designed a method with linear time and tiny constant memory that avoids regular expressions entirely. It produces results identical to the regex baseline. Microbenchmarks reportedly ran up to about two and a half times faster, and end-to-end encoding throughput raised by about fourteen percent depending on dataset. For phones and other edge hardware, such front-end savings compound across every generated response.
Catalog record
| CREATOR | Liu Zai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05833
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Peek2: A Regex-free implementation of pretokenizers for Byte-level BPE Liu Zai Abstract Pretokenization is a crucial, sequential pass in Byte-level BPE tokenizers. Our proposed new implementation, Peek2, serves as a drop-in replacement for cl100k-like pretokenizers used in GPT-3 4 , LLaMa-3 1 , and Qwen-2.5 12 . Designed with performance and safety in mind, Peek2 is Regex-free and delivers a 1.11 × 1.11\times improvement in overall throughput across the entire Byte-level BPE encoding process. This algorithm runs entirely on the CPU, has stable linear complexity O ( n ) O(n) , and provides presegmentation results identical to those of the original Regex-based pretokenizer. 1 Introduction Byte-level BPE tokenizers 11 are widely used in Large Language Models to transform raw text into a sequence of tokens. To overcome the limitations of the pair-merging process, which is usually slow and hard to parallelize, a pretokenizer is introduced to segment the original text into shorter fragments. These fragments then participate in the pair-merging process as individual inputs, enabling parallelization to achieve higher throughput 14 . However, the pretokenizer itself is usually implemented using complex Regular Expressions. This makes the process unsafe and inefficient. While versatile, using Regex to process text data raises several problems. The ideal time complexity for a DFA-based Regex engine 6 13 to execute an expression is O(n) , where n is the length of the input. But as the Regex expression get complex with multiple capture groups, it is found 3 7 that the time complexity may face regression to O(n\times k) or even O(n\times m) , where k is the number of groups and m is the length of pattern. If the patterns have the potential to fall back to each other, the DFA compilation gets complicated, and some implementations choose to use NFA instead, resulting in a worst time complexity of O(2^{n}) . Modern Regex engines support more advanced features like backreferences 10 2 , which makes the Regex execution an NP-Hard problem. The fact that Regex are arbitrary code loaded from files also creates new attacking surfaces 7 . Since the pair-merging algorithm cannot start until the pretokenizer finishes, optimizing throughput will improve the overall performance of the Byte-level BPE tokenizer encoding process. This will lead to performance gains across various tasks related to Large Language Models, such as preprocessing the training corpus, training the BPE tokenizer, and handling long contexts during inference. To address the above problems, we propose a new implementation of the widely adopted cl100k 9 pretokenizer, Peek2. Peek2 is Regex-free and has been tested to be more performant across all tokenizer-related tasks, thanks to the two new algorithms it introduces. Its improvements are thanks to the fact that not all Regex features are used, and that the aforementioned group fallback may not occur after a certain point. We found that only 7 categories of the first 2 characters have to be considered. 2 Background Cl100k 9 is the pretokenizer introduced with GPT-3 4 . It is widely adopted by other open source LLM tokenizers. The LLaMa-3 1 and Qwen-2.5 12 are examples of such models. The pretokenizer achieves the initial segmentation by applying the following regex pattern 9 repeatedly, inserting a break to split the original script at the end of every match: ’(?i:[sdmt]|ll|ve|re)| [^\r\n\p{L}\p{N}]?+\p{L}++| \p{N}{1,3}+| ?[^\s\p{L}\p{N}]++[\r\n]*+| \s++$|\s*[\r\n]|\s+(?!\S)|\s~ The linefolds are added for the ease of interpretation. The complete regex syntax rules and the subtle differences of its dialects are out of the scope of this work. The pretokenizer regex is formed in an or-of-groups hierarchy. The text is greedily matched against the leftmost group, and if the match fails, the next group is tested. The items like \p{} target a specific Unicode Character Class. 2.1 Left Snapping For convenience of writing, we also introduce the concept of Left Snapping . Take the following script of example: Lorem ipsum dolor sit amet. We use the | symbol to symbolize a break inserted in text, i.e., the following representation: Lorem| ipsum| dolor| sit| amet|. Splits the original script into 6 individual parts: • "Lorem" • " ipsum" • " dolor" • " sit" • " amet" • "." We notice that the space is combined with the next word to form a segment, contrary to the following Right Snapping example: Lorem |ipsum |dolor |sit |amet|. Anecdotally, Left Snapping is used to improve the tokenizer’s perception of Latin scripts. The exact reason for this convention is out of the scope of this work. 2.2 Branch0: Handling of Contractions The first group: (?i:[sdmt]|ll|ve|re) Matches common contractions in common Latin scripts. It has the highest priority, ensuring that these symbols are split out before any other patterns. 2.3 Branch1: Handling of Words The second group: [ˆ\r\n\p{L}\p{N}]?\p{L}+ Matches any group of letters, usually forming a word in Latin scripts, or a subsentence in East Asian scripts. The part before the question mark ? , ensures that it Left Snap to anything not of Unicode Letter Class, Unicode Number, ASCII CR, or ASCII LF. 2.4 Branch2: Handling of Numbers The third group: \p{N}{1,3} Demonstrates how numbers are handled. Unlike its predecessor, the r50k tokenizer 9 used in GPT-2, the numbers are segmented into groups of three digits. The following number: 12345678 Is segmented as: 123|456|78 The numbers do not Snap any characters. 2.5 Branch3: Handling of Punctuations and Others The fourth group: <space>?[ˆ\s\p{L}\p{N}]++[\r\n]*+ Has a leading space to be noticed. Anything not matching Unicode Letter Class, Unicode Number Class, or ASCII Whitespaces are grouped together. This cluster will also Left Snap one space, and Right Snap any count of line folds. 2.6 Branch4: Handling Of Whitespaces All 4 groups left are for handling of whitespaces: \s++$|\s*[\r\n]|\s+(?!\S)|\s Whitespaces are grouped together, with the following exceptions: • Will break after the last one of a cluster of line folds • Will break before the last whitespace, enabling the next word to left snap that whitespace if it is there 3 Peek2 We notice that, in the Regex implementation, the actual branch is determined completely by the first two Unicode scalars beyond the cursor, with only one exception: Branch0 may still fail after a mismatch on subsequent letter characters. We ignore this exception for now and will come back to it later. The idea of peeking two characters handles the aforementioned Left Snapping behavior as well, since the original implementation never Snaps more than one character. 3.1 Categorizing the Peeked Character We define PeekCategorize, which takes a Unicode scalar value and exclusively classifies its category. procedure PeekCategorize ( scalar ) if scalar=Space then return 1 else if scalar=SingleQuote then return 2 else if scalar=Cr|Lf then return 3 else category\leftarrow\textsc{UnicodeCategoryOf}(scalar) if category=Letter then return 4 else if category=Whitespace then return 5 else if category=Number then return 6 end if end if \textbf{return}{}0 end procedure Figure 1: PeekCategorize, classifying the peeked Unicode scalar value 3.2 Branch Decision As stated before, the 7\times 7 state space composed of the first two scalar categories decides the branch in only one look. So, we can list the corresponding branch in a 7\times 7 table, and in practice, provide a static lookup array of function pointers. Cat1 Cat0 0 1 2 3 4 5 6 0 3 3 3 3 1 3 3 1 3 4