Word Form Matters: LLMs’ Semantic Reconstruction under Typoglycemia
Chenxi Wang ★ Tianle Gu ★ Zhongyu Wei ♣ Lang Gao ★ Zirui Song ★ Xiuying Chen ★ 🖂 · 2025
Read Word Form Matters: LLMs’ Semantic Reconstruction under Typoglycemia by Chenxi Wang ★ Tianle Gu ★ Zhongyu Wei ♣ Lang Gao ★ Zirui Song ★ Xiuying Chen ★ 🖂 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01714.txt.
Introduction
Chenxi Wang and co-authors study how language models read scrambled words in this 2025 paper. People easily read words with jumbled middle letters, a quirk called typoglycemia, by leaning on word shape first and context second. Advanced models share the skill, but the inner mechanism was unclear.
The team built a score to measure how fully a model rebuilds the true meaning. Controlled tests showed word form, not context, does most of the work for models. Probing inside the models found special attention heads that extract word shape, and they stay steady as scrambling worsens. People, by contrast, shift flexibly between shape and context. The gap suggests ways to make models read more like humans.
Catalog record
| CREATOR | Chenxi Wang ★ Tianle Gu ★ Zhongyu Wei ♣ Lang Gao ★ Zirui Song ★ Xiuying Chen ★ 🖂 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01714
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Word Form Matters: LLMs’ Semantic Reconstruction under Typoglycemia Chenxi Wang ★ Tianle Gu ★ Zhongyu Wei ♣ Lang Gao ★ Zirui Song ★ Xiuying Chen ★ 🖂 Abstract Human readers can efficiently comprehend scrambled words, a phenomenon known as Typoglycemia , primarily by relying on word form; if word form alone is insufficient, they further utilize contextual cues for interpretation. While advanced large language models (LLMs) exhibit similar abilities, the underlying mechanisms remain unclear. To investigate this, we conduct controlled experiments to analyze the roles of word form and contextual information in semantic reconstruction and examine LLM attention patterns. Specifically, we first propose SemRecScore , a reliable metric to quantify the degree of semantic reconstruction, and validate its effectiveness. Using this metric, we study how word form and contextual information influence LLMs’ semantic reconstruction ability, identifying word form as the core factor in this process. Furthermore, we analyze how LLMs utilize word form and find that they rely on specialized attention heads to extract and process word form information, with this mechanism remaining stable across varying levels of word scrambling. This distinction between LLMs’ fixed attention patterns primarily focused on word form and human readers’ adaptive strategy in balancing word form and contextual information provides insights into enhancing LLM performance by incorporating human-like, context-aware mechanisms 1 1 1 https://github.com/Aurora-cx/TypoLLM . . 1 1 footnotetext: 🖂: Corresponding Author. 1 Introduction User: Do you undretsand Typoglycemia? LLM: Yes! Typoglycemia is a phenomenon where people can still read a word or sentence even when the middle letters of the words are scrambled, … Humans exhibit remarkable adaptability in reading, even when the internal character order of words is scrambled, as long as the first and last letters remain intact. This phenomenon, Typoglycemia , raises a fundamental question: Why do humans and LLMs understand scrambled words? Research shows that humans recognize words primarily through holistic shape matching Larson 2004 , relying on contextual prediction when shape cues are insufficient binti Ahmad Sabri 2015 . In extreme scrambling, higher-order regions like the prefrontal cortex enable retrospective reasoning to infer words from context Rayner et al. 2006 . Similarly, LLMs exhibit robustness to character scrambling. Cao et al. 2023 found they maintain high accuracy despite disrupted tokenization, while Yu et al. 2024 showed strong Typoglycemia task performance. These findings suggest LLMs, like humans, leverage contextual reasoning to reconstruct scrambled words. However, a key question remains unanswered: How do LLMs internally process scrambled text? Do they employ mechanisms similar to humans? To investigate this, we designed a series of controlled experiments to systematically analyze the effects of word form and contextual information on LLMs’ semantic reconstruction. We constructed a standardized dataset by carefully controlling key linguistic variables and evaluated LLaMA-3.2 (1B, 3B)-Instruct and LLaMA-3.3 (70B)-Instruct Dubey et al. 2024 to analyze their internal mechanisms underlying this capability. We define the Semantic Reconstruction Score (SemRecScore) as the cosine similarity between the representation of the original word’s token and the representation of the final subword token of the scrambled word at each layer of the LLM. This metric serves as a reliable measure of semantic reconstruction. Our results show that LLMs progressively recover word meaning across layers. However, as the degree of word form perturbation increases, semantic reconstruction quality gradually declines. At lower perturbation levels, words achieve near-complete semantic alignment with their original form, whereas at higher perturbation levels, only partial alignment is retained. In the final layers of the models, SemRecScores exhibit significant differences across different perturbation levels. In contrast, even when contextual information is entirely removed, its impact on semantic reconstruction remains minimal. Given that LLMs primarily rely on word form for semantic reconstruction, we further investigate how attention mechanisms facilitate this process. Our analysis reveals that attention allocation to word form follows a cyclic pattern across layers. Moreover, as word form perturbation increases, LLMs allocate progressively more attention to word form across all layers, suggesting that reconstructing highly scrambled words requires greater computational resources. Additionally, attention is not uniformly distributed; instead, LLMs rely on specific form-sensitive attention heads dedicated to processing word form information. As perturbation severity increases, more of these specialized attention heads are activated, yet their distribution remains stable, indicating a structured approach to leveraging word form information. These findings underscore a key distinction between LLMs and humans: while humans adaptively adjust their reliance on word form and contextual cues based on the degree of perturbation, LLMs primarily rely on word form and exhibit a relatively fixed yet structured attention allocation pattern. Our contributions can be summarized as follows: (1) We introduce a new perspective on LLM interpretability by investigating their internal mechanisms under character-level perturbations and propose SemRecScore as a reliable metric for quantifying LLMs’ semantic reconstruction ability. (2) Through systematic experiments, we demonstrate that word form is the primary factor in semantic reconstruction, while contextual information has minimal impact, and we reveal the key role of form-sensitive attention heads in this process. (3) We further uncover a fundamental divergence between LLMs and human cognition, providing insights for improving LLMs’ semantic adaptability. 2 Related Work Mechanisms Underlying Human Typoglycemia. Humans’ Typoglycemia ability suggests that word recognition does not strictly depend on letter order but involves higher-level cognitive mechanisms. Research in cognitive science and psychology suggests that word recognition primarily relies on word form rather than letter-by-letter decoding Shillcock et al. 2000 . Eye-tracking studies Johnson et al. 2007 ; White et al. 2008 show that fluent readers process words holistically rather than focusing on each letter. As long as the first and last letters remain intact, recognition remains stable Rayner et al. 2006 ; Johnson and Eisler 2012 . Beyond word shape, contextual priming further speeds up recognition Plummer et al. 2014 . For example, in “The nurse gave the patient a…”, words like ’doctor’ or ’medicine’ are automatically activated binti Ahmad Sabri 2015 , helping readers reconstruct scrambled words more efficiently. These findings suggest that human readers flexibly adapt between word form and context. LLM-Based Typoglycemia. Some recent studies have explored Typoglycemia and its impact on LLM robustness, but their analyses remain at a surface level, lacking a systematic investigation of internal mechanisms. For instance, Cao et al. 2023 demonstrated that GPT-4 and other advanced LLMs maintain strong language understanding under extreme character perturbations, accurately reconstructing scrambled words. However, their study was limited to input-output comparisons without examining how LLMs internally process scrambled text. Expanding on this, Yu et al. 2024 evaluated multiple LLMs on Typoglycemia tasks and found them remarkably robust. Yet, their focus remained on task-level performance rather than the underlying representations and attention mechanisms driving word recovery. Thus, while these works show that LLMs can handle Typoglycemia, they offer only observational insights. Our study fills this gap by analyzing LLMs’ internal processing under Typoglycemia perturbations. Robustness to Word Order Perturbations and Contextual Dependencies. Pham et al. 2021 and Sinha et al. 2021 found that BERT-based models maintain high accuracy in NLU tasks even when word order is shuffled, indicating a reliance on lexical co-occurrence patterns rather than syntactic structures. Similarly, Gupta et al. 2021 observed that BERT struggles to detect unnatural inputs, often making high-confidence predictions despite severe word order disruption. However, these studies focus on masked language models