Hallucinate or Memorize? The Two Sides of Probabilistic Learning in Large Language Models

Junichiro Niimi · 2025

Read Hallucinate or Memorize? The Two Sides of Probabilistic Learning in Large Language Models by Junichiro Niimi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08877.txt.

Description

Junichiro Niimi probes how language models memorize academic citations by asking GPT-4.1 to produce one hundred references across twenty computer science fields, then checking every field of every record against the truth. Citation counts serve as a stand-in for how redundantly a record appeared in training data.

Accuracy climbs with citation count in a log-linear way, with an inflection near ninety citations and saturation near twelve hundred, past which records come back nearly word-perfect. Memorization is layered: titles and first authors are recalled first, venues and numbers need far more repetition, and publication years are essentially never learned. Even famous records can be blended together when titles and authors overlap. The picture is of memory as a gradient, not a switch.

Catalog record

CREATORJunichiro Niimi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08877

Browse the full library · About the Tower of Babel