Are you going to finish that? A Practical Study of the Partial Token Problem

Hao Xu, Alisa Liu, Jonathan Hayase, Yejin Choi, Noah A. Smith · 2026

Read Are you going to finish that? A Practical Study of the Partial Token Problem by Hao Xu, Alisa Liu, Jonathan Hayase, Yejin Choi, Noah A. Smith (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23223.txt.

Introduction

Hao Xu, Alisa Liu, Jonathan Hayase, Yejin Choi, and Noah A. Smith study prompts that end mid-token. Language models are trained on token sequences, but users type raw text. When a prompt stops inside the model's expected next token, predictions can skew badly.

The authors call this the partial token problem and study it practically. It affects everyday tools, since people paste, truncate, and edit text mid-word all the time. Naming and measuring the flaw is the first step to interfaces that handle it gracefully. Distorted next-token predictions are the measurable harm, appearing whenever user text and model tokenization fall out of alignment.

Catalog record

CREATORHao Xu, Alisa Liu, Jonathan Hayase, Yejin Choi, Noah A. Smith
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-23223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are you going to finish that? A Practical Study of the Partial Token Problem Hao Xu, Alisa Liu, Jonathan Hayase, Yejin Choi, Noah A. Smith Abstract Language models (LMs) are trained over sequences of tokens, whereas users interact with LMs via text. This mismatch gives rise to the partial token problem , which occurs when a user ends their prompt in the middle of the expected next-token, leading to distorted next-token predictions. Although this issue has been studied using arbitrary character prefixes, its prevalence and severity in realistic prompts respecting word boundaries remains underexplored. In this work, we identify three domains where token and “word” boundaries often do not line up: languages that do not use whitespace, highly compounding languages, and code. In Chinese, for example, up to 25% of word boundaries do not line up with token boundaries, making even natural, word-complete prompts susceptible to this problem. We systematically construct semantically natural prompts ending with a partial tokens; in experiments, we find that they comprise a serious failure mode: frontier LMs consistently place three orders of magnitude less probability on the correct continuation compared to when the prompt is “backed-off” to be token-aligned. This degradation does not diminish with scale and often worsens for larger models. Finally, we evaluate inference-time mitigations to the partial token problem and validate the effectiveness of recent exact solutions. Overall, we demonstrate the scale and severity of probability distortion caused by tokenization in realistic use cases, and provide practical recommentions for model inference providers. 1 Introduction Users of language models (LMs) generally interface with LMs via text, providing a string of characters as prompt and expecting a string continuation as output. Under the hood, this is accomplished by using a tokenizer to segment text into a sequence of tokens , and prompting LMs trained over tokens to predict the next-token, which is then decoded back into text. While this generally occurs seamlessly and unbeknownst to the user, undesirable consequences of the mismatch appear when users (potentially unknowingly) provide a prompt that ends with the prefix of a valid token, which then causes the model to assign unexpectedly low probability to the completion of that token. This fundamental issue is known as the partial token problem (PTP; §2 ). 1 1 1 In other works, this issue has been called the “prompt boundary problem” or “tokenization bias,” but we choose to describe it as the partial token problem both to highlight the role of tokenization and to accurately reflect that the problem does not only occur at the prompt boundary. (See next footnote) It underlies (for example) the standard advice to users to not end prompts with a trailing whitespace, as the whitespace would be a partial token relative to the many tokens in the vocabulary that start with a whitespace. While the PTP has been extensively documented in prior work ( 20 ; 25 ; 23 ) and many methods have been proposed to address it ( 1 ; 32 ; 23 ; 31 ; 11 ) , these works generally illustrate the PTP using arbitrary character prefixes of the text; as a result, the extent to which the PTP affects natural use cases remains unknown. Figure 1 : Example model failures at performing a simple repeat-after-me task when given prompts that align with natural word boundaries but end with a partial token. Left: In Chinese, because 是一个 ( is a ) is a single token, the model does not generate the correct continuation “ 一个 ” ( a ) given “ 是 ” ( is ). Middle: In German, the model does not generate “ gelb ” ( yolk ) given “ ␣ Ei ” ( egg ). Right: In code, the model does not generate the colon “ : ” given “ ) ” because ): is a single token. Model continuations are shown for Qwen3-4B . In this work, we make the first attempt to demonstrate the severity of the PTP in natural and realistic prompts, including both its prevalence and the level of distortion it wreaks on next-token probability distributions. We identify three domains where whitespace does not reliably separate semantic or syntactic units (see illustrations in Fig. 1 ), where even prompts with complete word endings are susceptible ( §3 ). • Logographic writing systems. Unlike English, where words are typically separated by whitespace, words in languages such as Chinese and Japanese are not delimited. Tokenizers can therefore merge characters across word boundaries, and as a result, token and word boundaries often do not align. For instance, 是/一个 ( is / a ) is tokenized as the single token \langle{\color[rgb]{0.668,0.1992,0.4648}{\texttt{是一个}}}\rangle in the Qwen3 tokenizer, meaning that a prompt that ends with the complete word “ 是 ” ( is ) would be affected by the PTP. • Languages with highly productive compounding . In German, words are often built by combining two or more complete words. Thus, a tokenizer can segment compound words in ways that do not respect morphological boundaries. For example, Eigelb ( egg yolk ), which comes from Ei ( egg ) and gelb ( yellow ), is tokenized as \langle{\color[rgb]{0.668,0.1992,0.4648}{\texttt{\,\textvisiblespace\,E}}},{\color[rgb]{0.668,0.1992,0.4648}{\texttt{igel}}},{\color[rgb]{0.668,0.1992,0.4648}{\texttt{b}}}\rangle , so a prompt ending with “ ␣ Ei ” would similarly be affected by the PTP. • Code completion. Code often contains long identifiers or consecutive punctuation that are merged into single tokens. For example, in the Python main function definition “ def main(): ” , the final sequence of punctuation \langle{\color[rgb]{0.668,0.1992,0.4648}{\texttt{():}}}\rangle is a single token. As a result, the PTP affects prompts that ends with the punctuation “ () ” . Then, we systematically construct prompts that align with natural word boundaries but end with a partial token relative to the expected continuation, in order to create controlled experimental conditions that expose the PTP ( §4 ). We achieve this by constructing a “repeat-after-me” task in which the model is asked to repeat a given sentence with the target text partially provided. We find across a wide range of models that the PTP degrades model accuracy dramatically, by 60% – 95%. In Chinese, the probability assigned to the correct next-token drops by four orders of magnitude! Even more, this behavior does not improve with scaling. Finally, we investigate the effectiveness of proposed solutions to the PTP. Token healing ( 25 ) is a simple heuristic approach that “backs off” the prompt by removing a token from the prompt ending. More recently, some works have proposed exact solutions to the PTP ( 32 ; 23 ; 31 ; 11 ) , which claim to preserve the probability distribution of the LM at the level of text. For instance, 11 constructs a tree of all possible token sequences that “cover” the provided prompt, then samples a path from root to leaf. We find that this approach is effective, improving performance on the trivial repeat-after-me task to 100%, while heuristic approaches like token healing have more mixed performance. Overall, our work highlights how a fundamental problem caused by tokenization surfaces in realistic use cases, and provides new evaluations of a recent solution. 2 Background on partial token problem The task of language modeling is to define a probability P(\operatorname{{\color[rgb]{0.1914,0.4063,0.5586}continuation}}\mid\operatorname{{\color[rgb]{0.1914,0.4063,0.5586}prompt}}) for given \operatorname{{\color[rgb]{0.1914,0.4063,0.5586}prompt}} over all possible \operatorname{{\color[rgb]{0.1914,0.4063,0.5586}continuation}}

Read the complete text