Hallucinate or Memorize? The Two Sides of Probabilistic Learning in Large Language Models
Junichiro Niimi · 2025
Read Hallucinate or Memorize? The Two Sides of Probabilistic Learning in Large Language Models by Junichiro Niimi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08877.txt.
Description
Junichiro Niimi probes how language models memorize academic citations by asking GPT-4.1 to produce one hundred references across twenty computer science fields, then checking every field of every record against the truth. Citation counts serve as a stand-in for how redundantly a record appeared in training data.
Accuracy climbs with citation count in a log-linear way, with an inflection near ninety citations and saturation near twelve hundred, past which records come back nearly word-perfect. Memorization is layered: titles and first authors are recalled first, venues and numbers need far more repetition, and publication years are essentially never learned. Even famous records can be blended together when titles and authors overlap. The picture is of memory as a gradient, not a switch.
Catalog record
| CREATOR | Junichiro Niimi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08877