Splintering Nonconcatenative Languages for Better Tokenization
Bar Gazit βλ Shaltiel Shmidman δ Avi Shmidman χδ Yuval Pinter βλ β Department of Computer Science, λ Data Science Research Center χ Bar-Ilan University Ben-Gurion University of the Negev Ramat Gan, Israel Be’er Sheva, Israel δ DICTA, Jerusalem, Israel {bargazi@post,uvp@cs}.bgu.ac.il shaltieltzion@gm · 2025
Read Splintering Nonconcatenative Languages for Better Tokenization by Bar Gazit βλ Shaltiel Shmidman δ Avi Shmidman χδ Yuval Pinter βλ β Department of Computer Science, λ Data Science Research Center χ Bar-Ilan University Ben-Gurion University of the Negev Ramat Gan, Israel Be’er Sheva, Israel δ DICTA, Jerusalem, Israel {bargazi@post,uvp@cs}.bgu.ac.il shaltieltzion@gm (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14433.txt.
Introduction
Bar Gazit, Shaltiel Shmidman, Avi Shmidman, and Yuval Pinter fix a tokenizer flaw for root-based languages in a 2025 computing paper. Standard tokenizers slice text into chunks by simple string habits. Hebrew and Arabic build words by weaving roots into patterns, so naive slices cut through meaning. Their step, called SPLINTER, rearranges text first so the chunks a tokenizer finds line up with real word parts.
Tests covered Hebrew, Arabic, and Malay, both on the vocabularies produced and on full language models trained for Hebrew. Models fed the rearranged text learned better units and scored better downstream. The fix is a pre-step, so any common tokenizer can use it. Languages with woven morphology need not settle for tools built for English-style words.
Catalog record
| CREATOR | Bar Gazit βλ Shaltiel Shmidman δ Avi Shmidman χδ Yuval Pinter βλ β Department of Computer Science, λ Data Science Research Center χ Bar-Ilan University Ben-Gurion University of the Negev Ramat Gan, Israel Be’er Sheva, Israel δ DICTA, Jerusalem, Israel {bargazi@post,uvp@cs}.bgu.ac.il shaltieltzion@gm |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14433
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Splintering Nonconcatenative Languages for Better Tokenization Bar Gazit βλ Shaltiel Shmidman δ Avi Shmidman χδ Yuval Pinter βλ β Department of Computer Science, λ Data Science Research Center χ Bar-Ilan University Ben-Gurion University of the Negev Ramat Gan, Israel Be’er Sheva, Israel δ DICTA, Jerusalem, Israel {bargazi@post,uvp@cs}.bgu.ac.il [email protected] [email protected] Abstract Common subword tokenization algorithms like BPE and UnigramLM assume that text can be split into meaningful units by concatenative measures alone. This is not true for languages such as Hebrew and Arabic, where morphology is encoded in root-template patterns, or Malay and Georgian, where split affixes are common. We present Splinter , a pre-processing step which rearranges text into a linear form that better represents such nonconcatenative morphologies, enabling meaningful contiguous segments to be found by the tokenizer. We demonstrate Splinter ’s merit using both intrinsic measures evaluating token vocabularies in Hebrew, Arabic, and Malay; as well as on downstream tasks using BERT-architecture models trained for Hebrew. 1 Introduction Large language models (LLMs) have become pivotal in natural language processing (NLP), offering extensive utility across diverse applications. Central to constructing an LLM is producing basic input units from the text sequence, for which subword tokenization is still the standard approach, using methods such as byte-pair encoding (BPE) Sennrich et al. (2016) , WordPiece Schuster and Nakajima (2012) , and UnigramLM Kudo (2018) . However, subword tokenizers exhibit diminished effectiveness in nonconcatenative languages (NCLs) such as Hebrew and Arabic Klein and Tsarfaty (2020) . While tokenizers assume linear segmentation of words, NCLs’ units of meaning are typically intertwined within words, and cannot be separated linearly , as the root letters are not adjacent Khaliq and Carroll (2013) . A Hebrew example is provided in Table 1 . in the Hebrew word l‘bwd! la’avod ‘to work’, the root letters are “!’, ‘b!’, and ‘d!’, placed in an infinitive morphological template manifested by the locations of ‘l!’ and ‘w!’. This characteristic forces linear tokenizers to split words into morphologically-incoherent tokens, losing the downstream models’ ability to generalize across various forms of the same lemma, and eventually reducing model performance when applied to a large variety of tasks such as text generation and translation Keren et al. (2022) ; Levi and Tsarfaty (2024) ; Shmidman et al. (2024) . (a) .‘bdty ‘l hm.sgt! ‘I worked on the presentation.’ (b) .h‘bwdh hq/sh h/stlmh! ‘The hard work paid off.’ Table 1: Examples of Hebrew text (read from right to left) exemplifying its nonconcatenative morphology. The root ‘bd! ‘work’ appears in both sentences, but in (a) it comprises a linear segment of the text whereas in (b) it is broken by the templatic character w!. We present Splinter , a statistical algorithm for linearizing NCL text through rearranging the text sequence by iteratively pruning characters from words, with the intent of isolating characters representing template forms. The manipulated text can then be input into any ordinary linear tokenizer for processing as usual, adapting the NCL data into the morphologically-concatenative input BPE and its like expect. We show that vocabularies and models trained over Splinter -processed text outperform those starting from raw NCL text on both intrinsic and extrinsic measures in Hebrew, Arabic, and Malay. 1 1 1 https://github.com/MeLeLBGU/Splintering . 2 Tokenizing with Splinters Standard Flow: ‘d!’ ‘w!’ ‘b!’ “!’ ‘l!’ Original text Input ‘bwd!’ ‘l‘!’ Segmented text Tokenizer LM Embedding Splinter Flow: ‘d!’ ‘w!’ ‘b!’ “!’ ‘l!’ Original text Input ‘3:w!’ ‘0:l!’ ‘d!’ ‘b!’ “!’ Splintered text Splinter ‘3:w! 0:l!’ “bd!’ Segmented text Tokenizer LM Embedding Figure 1: Overview of a Hebrew language model pipeline using the example ‘to work’: standard flow vs. incorporating Splinter . This figure is a real example for the differences in tokenization using a BPE tokenizer with a vocab size of 2000. Gray boxes are ordered from right to left. When designing our approach, our goal was to create a relinearized sequence for words in NCL languages, while adhering to several constraints. One constraint is that the transformation must be reversible, ensuring that the new representation can always be converted back to the original text. In addition, we aimed to develop a tool that would integrate smoothly with existing tokenizers, ensuring seamless adoption without requiring modifications in their implementation. Additionally, the method should be applied only to the intended languages, without affecting the entire text. We also considered that the method should be adaptable to two distinct use cases: (1) models trained primarily in an NCL language, where the majority of the vocabulary belongs to that language (e.g., DictaBERT ( Shmidman et al., 2023 ) has a large vocabulary size of 128K); and (2) large-scale multilingual LLMs, where most tokens are allocated to English and only a small portion is left for the NCL language (e.g., GPT-4o, which has approximately 2.3K tokens allocated for Hebrew). Thus, the method should be effective across different vocabulary sizes. We develop Splinter as a pre-tokenization step designed to address the challenges of subword tokenization in NCLs. The core idea of the algorithm is that in certain NCLs, many words are formed by embedding root letters into specific morphological templates. For instance, the Hebrew word l‘bwd! is derived from the root ‘bd! placed in the template l__w_!. Since there are far fewer templates than roots, template letters tend to appear in specific positions within a word more consistently than root letters do (e.g., for l‘bwd!: “0:l!”,“3:w!”). Empirically, we observed that in Hebrew and Arabic, when a word is longer than 3 characters, there is always at least one deletion that, when applied, transitions the word to a different, existing template while preserving the same root . 2 2 2 However, not every letter’s deletion will necessarily result in a valid template. For example, removing d! from l‘bwd! would produce the non-word l‘bw!. For instance, the word l‘bwd! turns into l‘bd! when the template letter w! is removed. By repeating this process iteratively, the word eventually reduces to only its root letters. This method can be seen as a way to isolate the root letters from the template. In this method, NCL words are transformed into a sequence of single-letter reductions, with the goal of rearranging them to better align with existing subword tokenizers. This is achieved by expanding the language’s alphabet to include not only its original letters but also single-letter reductions, where each reduction consists of a letter paired with the index from which it was removed. 3 3 3 In practice, we found that using negative indices for the latter half of a word’s characters both aligns well with the suffixing nature of certain morphemes and reduces the resulting alphabet by 15%. Not shown in our examples for simplicity. Further low-level details are provided in Appendix A . From another perspective, this method explores the impact of expanding a language’s alphabet to enhance word representation. We illustrate the high-level application of Splinter into the NLP pipeline in Figure 1 . 2.1 Splinter Reduction Map Creation As mentioned above, the Splinter pre-tokenization step processes a word by iteratively applying single-letter reductions . To achieve this, we must determine which reduction to perform at each iteration. We propose an algorithm that analyzes a given corpus and generates a mapping, where the keys represent word lengths and the