Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data

Bao Pham, Mohammed J. Zaki, Luca Ambrogioni, Dmitry Krotov, Matteo Negri · 2026

Read Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data by Bao Pham, Mohammed J. Zaki, Luca Ambrogioni, Dmitry Krotov, Matteo Negri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26841.txt.

Description

Pham and four colleagues show that uniform-based discrete diffusion language models behave as associative memories, systems that recover stored items through basins of attraction. As training data grows, basins around training examples shrink while basins around unseen test examples expand, until both converge. The team detects this memorization-to-generalization shift using only the conditional entropy of predicted token sequences, which vanishes under memorization and stays finite under generalization. Conditional entropy thus offers a practical probe for deployed models.

Catalog record

CREATORBao Pham, Mohammed J. Zaki, Luca Ambrogioni, Dmitry Krotov, Matteo Negri
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54798
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26841

Browse the full library · About the Tower of Babel