Guess or Recall? Training CNNs to Classify and Localize Memorization in LLMs
Jérémie Dentan, Davide Buscaldi, Sonia Vanier · 2025
Read Guess or Recall? Training CNNs to Classify and Localize Memorization in LLMs by Jérémie Dentan, Davide Buscaldi, Sonia Vanier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02573.txt.
Description
Jérémie Dentan, Davide Buscaldi, and Sonia Vanier probe how language models memorize text word for word, in a paper from twenty twenty-five. A model may repeat a passage because it truly stored it, or because the passage is easy to guess. Sorting these cases matters for privacy and for science.
The team trained image-style networks on the model's internal attention maps to sort samples by type. The old sorting scheme matched those maps poorly. Their new scheme has three kinds: guessed samples, samples recalled after heavy repetition in training, and non-memorized ones. Much of what looks extractable turns out to be mere guessing. A custom visual tool pinpoints which attention regions serve each kind.
Catalog record
| CREATOR | Jérémie Dentan, Davide Buscaldi, Sonia Vanier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-02573