Spelling Bee Embeddings for Language Modeling

Markus N. Rabe, Judith Clymo, Zheren Dong · 2026

Read Spelling Bee Embeddings for Language Modeling by Markus N. Rabe, Judith Clymo, Zheren Dong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18030.txt.

Introduction

Markus N. Rabe, Judith Clymo, and Zheren Dong give language models a sense of spelling. Their 2026 preprint changes the embedding layer, the part that turns tokens into numbers. The change feeds each token's letters into its embedding. Models trained this way do better not only on spelling tasks but across standard tests as well. The team runs scaling studies with models from forty million to eight hundred million parameters.

Spelling is usually invisible to a model that sees only whole tokens. That blindness hurts on names, rare words, and typos. Letter-aware embeddings attack the problem at the root. The gains across broad tests suggest the fix helps general skill, not just spelling bees.

Catalog record

CREATORMarkus N. Rabe, Judith Clymo, Zheren Dong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18030

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spelling Bee Embeddings for Language Modeling Markus N. Rabe, Judith Clymo, Zheren Dong Abstract We introduce a simple modification to the embedding layer. The key change is to infuse token embeddings with information about their spelling. Models trained with these embeddings improve not only on spelling tasks, but also across standard benchmarks. We conduct scaling studies for models with 40M to 800M parameters, which suggest the improvements are equivalent to needing about 8% less compute and data to achieve the same test loss. The improvements also translate to gains on a wide set of benchmarks. 1 Introduction The question to “count the number of Rs in strawberry” is a popular example that demonstrates the unintuitive failure modes of language models. It is a seemingly trivial question that tripped up even the largest language models, which insisted that strawberry contains just two Rs. The failure is widely attributed to tokenization, i.e., the process of grouping characters in a text into tokens before applying a language model. To develop an intuition about tokenization challenges, let’s consider how GPT-4 tokenizes variants of the word strawberry: strawberry str aw berry ␣strawberry strawberry Strawberry Str aw berry ␣Strawberry Strawberry (strawberry (str aw berry (Strawberry (Str aw berry ,strawberry ,str aw berry =strawberry =str aw berry STRAWBERRY ST RAW B ERRY ’STRAWBERRY ’S TR AW B ERRY ’strawberry ’s tr aw berry .Strawberry .Str aw berry .strawberry .str aw berry [strawberry [str aw berry -strawberry -str aw berry _strawberry _str aw berry We observe that tokenization depends heavily on leading spaces, surrounding special characters, and capitalization. In total, 22 distinct tokens are involved in the variants above. And these examples above do not even consider plural forms, misspellings, and splitting the word due to line breaks etc. Any language model must learn to associate all these token sequences with the same underlying word and spelling. This is not an isolated case. The word "apple" occurs 11 times in GPT4’s vocabulary: ‘ Apple’, ‘apple’, ‘ apple’, ‘Apple’, ‘.apple’, ‘ apples’, ‘ AppleWebKit’, ‘/apple’, ‘ pineapple’, ‘APPLE’, ‘ APPLE’. These variants need to have different tokenizations, such that the model can distinguish them, but it also means that models need to learn the word’s spelling many times over. Languages with heavier use of inflections suffer even more from tokenization artifacts. Consider the variants of the German word “hoch” (=high): hoch, hohe, hohem, hohen, hoher, hohes, höher, höhere, höherem, höheren, höherer, höheres, höchst, höchste, höchstem, höchsten, höchster, and höchstes. Their tokenizations make use of 19 distinct tokens—and this does not even account for capitalization, leading spaces or special characters, or compound words. While language models can learn about the spelling of tokens with enough data, we argue that this makes the learning process less efficient . Indeed, even models with hundreds of billions of parameters trained on trillions of tokens struggle to reliably count the number of Rs in strawberry. While many models have improved on the canonical ‘strawberry’ prompt over time, even small variants trip up recent models. We present some examples in Appendix A . strawberry byte embedding for s byte embedding for t byte embedding for r … A spelling lookup position-aware s position-aware t position-aware r … A RoPE sum & scale spelling embedding token embedding e_{\text{tok}} lookup in embedding table spelling bee embedding average Figure 1: Spelling bee embeddings augment a standard token embedding e_{\text{tok}} with a normalized sum of byte-level embeddings for the first 16 bytes of the token. Byte embeddings are position-encoded with RoPE using character positions inside the token. In this paper, we propose spelling bee embeddings , a simple architectural change that helps language models with the spelling of tokens. Spelling bee embeddings introduce a negligible amount of additional parameters and essentially no computational overhead. Compared to a well-tuned baseline, spelling bee embeddings consistently improve the performance of models from 40m parameters to 800m parameters. We also confirm that models trained with spelling bee embeddings can determine the number of Rs in strawberry. Like regular token embeddings, spelling bee embeddings have 1 embedding per token, but we also introduce 256 new embeddings, one for each possible value of a byte. The output of spelling bee embeddings is the addition of the token embedding and the embeddings of the first 16 bytes of the token. We further improve this "bag of characters" representation by applying rotary position embeddings to the characters, using their position in the token instead of the sequence position. 2 Related Work 13 compare the performance of language models to humans on spelling challenges, and find that language models indeed struggle on these tasks compared to humans. 4 present detailed statistics on spelling challenges in LLMs and suggest that spelling challenges are not related to the frequency of words in the training data. There have been many attempts to make models aware of character-level structure, e.g. by removing the need for tokenization from language modeling. Most notably are language models that operate on the level of bytes. Character-level Transformers (without tokenization) can match or exceed token-based models given the same data. However, this comes at a dramatically increased computational cost, as the sequence length increases significantly for the same number of text characters. Some efforts have been made to learn token-like groupings inside the model and thereby keep the computation cost at bay. Notable examples are Charformer ( 16 ) and MegaByte ( 21 ) . However, these methods didn’t control for sequence length and compute cost. 8 and 9 proposed variants of transformers that learn to segment the text in an end-to-end fashion. While their work is beautiful and may eventually obsolete tokenization (and spelling bee embeddings) alltogether, they are also more risky to adopt for existing large training runs due to their complexity. In contrast, spelling bee embeddings keep the changes to the architecture very local to the embedding layer, which makes them compatible with language models that don’t follow the Transformer architecture and greatly simplify their adoption. Spelling bee embeddings also work well with existing tokenizers, so that existing handling of special tokens etc does not have to change. This also enables a clean apples-to-apples comparison, as the compute cost and sequence length are unaffected. 3 Spelling Bee Embeddings The core idea behind spelling bee embeddings is to make model embeddings aware of the spelling of tokens, without sacrificing the computational advantages of subword tokenization. Instead of relying on the model to infer character-level information implicitly, we augment the token embeddings with a compact representation of the token’s spelling. Formally, for each token ID t in the vocabulary, we introduce: • A learned token embedding e_{\text{tok}}(t)\in\mathbb{R}^{d} , as in standard models; • A sequence of at most 16 bytes \{b_{1},b_{2},\dots,b_{16}\} obtained from its UTF-8 encoding. If the token’s UTF8 encoding is shorter than 16 bytes, we pad with the null byte; if longer, we truncate. Each byte b_{i}\in\{0,\dots,255\} is embedded through a learned lookup table of size 256\times d , producing byte embeddings \{e_{b_{1}},\dots,e_{b_{16}}\} . To distinguish the contribution of each character within the token, we apply rotary positional embeddings ( 15 ) to the byte embeddings, using the position of each byte inside the token: \tilde{e}_{b_{i}}=\text{RoPE}(e_{b_{i}},i). We then compute the combined character embedding for

Read the complete text