Spelling-out is not Straightforward: LLMs’ Capability of Tokenization from Token to Characters
Tatsuya Hiraoka, Kentaro Inui · 2025
Read Spelling-out is not Straightforward: LLMs’ Capability of Tokenization from Token to Characters by Tatsuya Hiraoka, Kentaro Inui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10641.txt.
Introduction
Tatsuya Hiraoka and Kentaro Inui study how large language models spell out tokens character by character, in work from 2025. Models do this task with high accuracy, yet they fail at harder character-level tasks, such as finding parts inside a token. The authors ask how character data is stored and used inside the model during spelling.
The embedding layer does not fully hold character data, most clearly past the first character. Later Transformer layers rebuild that knowledge, with a clear breakthrough point in spelling behaviour. Three checks support this account: probing classifiers, knowledge neuron tests and attention weight inspection. The work explains a skill that looks simple but is not handled in a simple way.
Catalog record
| CREATOR | Tatsuya Hiraoka, Kentaro Inui |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-10641
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spelling-out is not Straightforward: LLMs’ Capability of Tokenization from Token to Characters Tatsuya Hiraoka, Kentaro Inui Abstract Large language models (LLMs) can spell out tokens character by character with high accuracy, yet they struggle with more complex character-level tasks, such as identifying compositional subcomponents within tokens. In this work, we investigate how LLMs internally represent and utilize character-level information during the spelling-out process. Our analysis reveals that, although spelling out is a simple task for humans, it is not handled in a straightforward manner by LLMs. Specifically, we show that the embedding layer does not fully encode character-level information, particularly beyond the first character. As a result, LLMs rely on intermediate and higher Transformer layers to reconstruct character-level knowledge, where we observe a distinct “breakthrough” in their spelling behavior. We validate this mechanism through three complementary analyses: probing classifiers, identification of knowledge neurons, and inspection of attention weights. 1 Introduction While large language models (LLMs) have grown remarkably in recent years, several studies report that they still struggle with fine-grained character-level manipulations, such as inserting, deleting, or extracting individual characters within tokens Edman et al. 2024 ; Wang et al. 2024 ; Chai et al. 2024 ; Shin and Kaneko 2024 . Although most LLMs operate over subword tokens, true mastery of subtoken information is essential for a range of applications, such as morphological inflection Marco and Fraser 2024 , letter counting Fu et al. 2024 , typoglycemia Wang et al. 2025 , and handling typos Tsuji et al. 2025 . To improve their reliability in such scenarios, we must understand how LLMs internally represent and process characters. A paradox emerges from prior work: LLMs can accurately spell out entire tokens as sequences of characters Edman et al. 2024 ; Xiong et al. 2025 , while they often fail at simple tasks such as identifying a single character at a fixed position Hiraoka and Okazaki 2024 ; Chai et al. 2024 . For instance of a token “ language ,” models readily generate “ l a n g u a g e ” on demand but cannot reliably extract “ u ” at position five. This discrepancy suggests that, despite lacking explicit access to compositional character knowledge, LLMs have some mechanism for character-by-character spelling. Figure 1: Summary of our findings. LLMs spell out tokens by relying directly on embedding-level information for the first character, but gradually shift to using distributed, higher-layer representations for later characters. We found a “breakthrough” layer where character knowledge becomes detectable. In this paper, we investigate how and where LLMs capture and deploy character‐level knowledge during spelling-out. We begin by constructing a token-characters dataset from the vocabularies of four representative LLMs (§ 3 ) and confirming their ability to spell out tokens with few‐shot prompts (§ 4 ). Probing the embedding layer reveals that it does not encode subtoken characters directly (§ 5 ), but downstream Transformer layers gradually recover this information. We identify a clear “breakthrough” layer at which character identities become reliably detectable by our probing classifier (§ 6 ), and observe that the first character is handled differently from subsequent ones. Finally, through neuron‐level analyses (§ 7 ) and attention‐weight inspections (§ 8 ), we trace how character knowledge is stored and routed, demonstrating that its locus aligns precisely with the breakthrough point. Taken together, our study sheds new light on the internal character-level machinery of LLMs. Our main contributions are as follows: • We demonstrate that token embeddings do not fully encode character-level information, and that subsequent Transformer layers play a major role in reconstructing this information during the spelling-out process. • We pinpoint the exact layer where compositional character information “breaks through,” validated via probing classifiers. • We investigate character knowledge to specific neurons and attention patterns, showing alignment with the breakthrough layer. • We provide a diagnostic framework (dataset, probes, and analyses) for future research into subtoken and character-level modeling. 2 Related Work Our research is in line with work focusing on the LLMs’ capability of character-level manipulation. Edman et al. 2024 introduced a dataset to evaluate this capability from some viewpoints of character-level manipulation, including spelling-out of words or tokens. Similarly, Wang et al. 2024 provided a dataset of complicated character-level manipulation tasks. Both works conclude that LLMs have limited capability for complicated character processing. Furthermore, these works reported that LLMs can spell out words in the original order, as reported in Xiong et al. 2025 , while they do not have sufficient knowledge of their single characters inside words Shin and Kaneko 2024 ; Hiraoka and Okazaki 2024 ; Chai et al. 2024 unless fine-tuning for the models to learn the token internal information directly Xu et al. 2024 . This line of literature motivates us to investigate the LLMs’ internal workings of spelling-out behavior, despite the lack of character-level knowledge. Recently, we can see a trend of understanding LLMs’ capability of recognizing character-level information, such as the ability of counting characters Fu et al. 2024 . Moreover, beyond the word-to-character spelling out, Wu et al. 2025 investigates their ability to recognize radicals inside Chinese characters. Our work is also related to the findings of LLMs’ “detokenization” ability in their later layers Kaplan et al. 2025 ; Kamoda et al. 2025 , which is an ability to internally merge tokens into words or phrases. In contrast, we focus on the inverse problem: how LLMs internally “tokenize” tokens into characters. 3 Experimental Setup 3.1 Target Models We investigate four medium-sized LLMs ( \approx 7 B parameters): LLaMA3-8B Dubey et al. 2024 , Gemma-7B Team et al. 2024 , Qwen2.5-7B Yang et al. 2024 , and Amber Liu et al. . We selected them because our preliminary trials showed that smaller models (e.g., 3B) fail to spell out tokens with sufficient accuracy. All models have the Transformer-based architecture Vaswani et al. 2017 . Table 1 shows each model’s vocabulary size, which ranges from 32K to 256K tokens. 3.2 Evaluation Dataset # Vocab # Dataset % LLaMA3-8B 128,256 19,724 15.38 Gemma-7B 256,000 47,833 18.68 Qwen2.5-7B 152,064 18,973 12.48 Amber-6.7B 32,000 6,130 19.16 Table 1: The number of tokens in the vocabulary of each LLM ( # Vocab ) and in our dataset ( # Dataset , § 3.2 ). % shows the ratio of in-dataset tokens in the vocabulary. Our evaluation focuses on spelling out single tokens into their constituent characters. In other words, we exclude multi-token words to ensure a well-controlled experimental setting. For example, in the case of “ token/s ”, the model could easily reveal the final “ s ” without requiring character-level understanding. We construct the dataset from each model’s vocabulary by selecting all single tokens that: 1) contain only lowercase alphabets ( a-z ), 2) begin with the special prefix indicating the head of words (e.g., “ _ ” in “ _token ”), and 3) are at least five characters. This yields a set comprising roughly 15 % of each model’s full vocabulary (Table 1 ). Note that some tokens (e.g., “somew,” “immedi”) are subword fragments rather than complete words. Figure 9 and 10 in the appendix show the distribution of frequency of token length and alphabet, respectively. Given the target tokens in our dataset, LLMs