Exploring Approaches for Detecting Memorization of Recommender System Data in Large Language Models

Antonio Colacicco, Vito Guida, Dario Di Palma, Fedelucio Narducci, Tommaso Di Noia · 2026

Read Exploring Approaches for Detecting Memorization of Recommender System Data in Large Language Models by Antonio Colacicco, Vito Guida, Dario Di Palma, Fedelucio Narducci, Tommaso Di Noia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-02002.txt.

Description

Language models are trained on undisclosed mountains of text, so private or benchmark data may hide inside them. Antonio Colacicco and four co-authors probe a known leak. The MovieLens-1M dataset is memorized by LLaMA models. Can it be extracted by machine? The alternative is hand-crafted prompts.

Three strategies were tried. Jailbreak prompting stayed erratic. Probing internal activations separated real from fake movie titles. It failed on numbers. Automatic prompt engineering refines prompts in rounds. It recovered item-level information moderately well. Automatic prompt optimization looks most promising, though numerical interaction data resisted all three approaches.

Catalog record

CREATORAntonio Colacicco, Vito Guida, Dario Di Palma, Fedelucio Narducci, Tommaso Di Noia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-02002

Browse the full library · About the Tower of Babel