Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms

Yuto Nishida Naoki Shikoda Yosuke Kishinami Ryo Fujii, Makoto Morishita Hidetaka Kamigaito Taro Watanabe · 2026

Read Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms by Yuto Nishida Naoki Shikoda Yosuke Kishinami Ryo Fujii, Makoto Morishita Hidetaka Kamigaito Taro Watanabe (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21882.txt.

Introduction

Nishida and six colleagues examine non-verbatim memorisation in large language models through entity surface forms. Typical entity-based question answering uses one canonical name per entity, mixing fact memorisation with access through that name. Their RedirectQA dataset links Wikidata factual triples to redirect-based surface forms, including alternative names, abbreviations, spelling variants, and common erroneous forms. Across 13 models, outcomes often change when only the surface form changes. Models handle minor spelling variations better than aliases and abbreviations, and entity frequency contributes beyond surface frequency.

Catalog record

CREATORYuto Nishida Naoki Shikoda Yosuke Kishinami Ryo Fujii, Makoto Morishita Hidetaka Kamigaito Taro Watanabe
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55364
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21882

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms Yuto Nishida Naoki Shikoda Yosuke Kishinami Ryo Fujii, Makoto Morishita Hidetaka Kamigaito Taro Watanabe Abstract Understanding what kinds of factual knowledge large language models (LLMs) memorize is essential for evaluating their reliability and limitations. Entity-based QA is a common framework for analyzing non-verbatim memorization, but typical evaluations query each entity using a single canonical surface form, making it difficult to disentangle fact memorization from access through a particular name. We introduce RedirectQA , 1 1 1 https://huggingface.co/datasets/naist-nlp/RedirectQA an entity-based QA dataset that uses Wikipedia redirect information to associate Wikidata factual triples with categorized surface forms for each entity, including alternative names, abbreviations, spelling variants, and common erroneous forms. Across 13 LLMs, we examine surface-conditioned factual memorization and find that prediction outcomes often change when only the entity surface form changes. This inconsistency is category-dependent: models are more robust to minor orthographic variations than to larger lexical variations such as aliases and abbreviations. Frequency analyses further suggest that both entity- and surface-level frequencies are associated with accuracy, and that entity frequency often contributes beyond surface frequency. Overall, factual memorization appears neither purely surface-specific nor fully surface-invariant, highlighting the importance of surface-form diversity in evaluating non-verbatim memorization. Surface Category Type Question LLM Answer Eval. Case 1 Canonical - What is David Guetta ’s occupation? DJ ✓ from pseudonyms Alt./Abbrev. What is Jack Back ’s occupation? actor ✗ Case 2 Canonical - What sport does José García Castro play? baseball ✗ from alternative names Alt./Abbrev. What sport does Pepillo II play? soccer ✓ Case 3 Canonical - In what city was J. M. Coetzee born? Cape Town ✓ from modifications Spell. Var. In what city was J M Coetzee born? Cape Town ✓ Case 4 Canonical - What is Stan Coveleski ’s occupation? baseball player ✓ from misspellings Typ. Err. What is Stan Covaleski ’s occupation? actor ✗ Table 1: Illustrative examples of surface-conditioned factual access in RedirectQA. Each pair of rows refers to the same Wikidata entity and factual triple; only the subject entity surface form in the question is changed. The gold answer is therefore fixed within each case, but the Pythia-12B predictions can flip between ✓ correct and ✗ incorrect. The examples show canonical-to-redirect failures, the reverse pattern, robustness to a minor orthographic variant, and fragility to a common misspelling. Aggregate results across 13 LLMs are reported in section 3 . 1 Introduction Large language models (LLMs) store a wide range of factual knowledge in their parameters, enabling them to answer many knowledge-intensive questions without external retrieval 22 ; 30 . At the same time, when the required knowledge is absent or inaccessible, LLMs may produce hallucinated or erroneous answers 27 . Understanding what factual knowledge LLMs memorize non-verbatim, and under what conditions they can access it, is therefore central to evaluating their reliability and limitations. A common way to analyze non-verbatim memorization is entity-based question answering (QA), where models are queried about factual relations involving entities and memorization is measured by answer accuracy 26 ; 16 ; 15 . This line of work has shown that facts about low-frequency or low-popularity entities are less likely to be memorized 13 ; 16 ; 15 . However, typical evaluations instantiate each entity using a single canonical surface form. This makes it difficult to disentangle whether a model has memorized a fact about an entity from whether it can access that fact through the particular name used in the question. This distinction matters because entities are often referred to by multiple surface forms. A model that answers correctly for a canonical name such as Pelé may not necessarily access the same fact when the entity is referred to as Edson Arantes do Nascimento . Indeed, in our preliminary diagnostic using Pythia-12B 2 on a redirect-augmented version of PopQA 16 , 23.7% of canonical–redirect question pairs yield inconsistent predictions (Appendix B.1 ). This observation motivates a systematic evaluation in which the underlying fact is controlled while the entity surface form is varied. To analyze this phenomenon systematically, we introduce RedirectQA , an entity-based QA dataset that associates Wikidata factual triples with multiple entity surface forms using Wikipedia redirect information. The key design of RedirectQA is to hold the factual relation and gold answer fixed while varying only the surface form of the subject entity. As illustrated in table 1 , this design exposes cases where a model answers correctly under one surface form but incorrectly under another, even though the underlying fact is unchanged. Redirect surface forms are further annotated with categories such as alternative names, abbreviations, spelling variants, and common erroneous forms, enabling controlled analyses of how different types of naming variation affect factual QA. Using RedirectQA, we evaluate 13 LLMs and find that prediction outcomes often differ across surface forms of the same entity, even though the underlying factual triple is held fixed. The inconsistency is category-dependent: models are relatively robust to minor orthographic variations, such as spelling differences, diacritics, and punctuation changes, but are less consistent for larger lexical variations, such as aliases, alternative names, and abbreviations. These results indicate that non-verbatim memorization cannot be treated as fully surface-invariant, even when the entity and fact remain the same. We further analyze how entity- and surface-level frequencies relate to memorization. By decomposing aggregate entity frequency into surface-level frequencies, we find that accuracy is associated with both the frequency of a specific surface form and the aggregate frequency of the corresponding entity, with entity frequency often contributing beyond surface frequency. This pattern suggests cross-surface coupling in factual access, rather than purely independent memorization of each surface form. Together with the consistency results, these findings point to an intermediate picture in which factual memorization is neither purely surface-specific nor fully surface-invariant. Overall, our work shows that evaluating non-verbatim memorization through canonical entity names alone can miss surface-conditioned failures in factual access. RedirectQA provides a controlled resource for studying these effects, highlighting surface-form diversity as a key factor in evaluating what LLMs memorize and how reliably they can access it. 2 RedirectQA We introduce RedirectQA , an entity-based factual QA dataset designed to analyze how LLMs access the same factual knowledge through different surface forms of an entity. RedirectQA associates Wikidata factual triples in the form of (\textit{subject},\textit{relation},\textit{object}) with multiple subject entity surface forms using Wikipedia redirect information. The key design is to keep the factual relation and gold answer fixed while varying the surface form of the subject entity. We follow the open-domain QA setting ( 24 ) , evaluating models on factual questions without providing external evidence. 2.1 Wikipedia Redirects as Surface-Form Resources Wikipedia article titles are chosen according to naming guidelines, 2 2 2 https://en.wikipedia.org/wiki/Wikipedia:Article_titles typically favoring recognizable, natural, and searchable expressions among possible names for a topic or entity. To make articles accessible through alternative expressions, Wikipedia provides redirect pages, which automatically forward users from a redirect title to the corresponding main article. For example, the page titled “NYT” redirects to the article “The New York Times.” Such redirects provide a large-scale source of surface forms that refer to the same underlying entity. Redirect pages are often annotated with redirect categories that describe the relationship between the

Read the complete text