In-context denoising with one-layer transformers: connections between attention and associative memory retrieval

Matthew Smart, Alberto Bietti, Anirvan M. Sengupta · 2025

Read In-context denoising with one-layer transformers: connections between attention and associative memory retrieval by Matthew Smart, Alberto Bietti, Anirvan M. Sengupta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05164.txt.

Description

Matthew Smart, Alberto Bietti, and Anirvan Sengupta show that a single attention layer can clean a noisy pattern the way a memory network does. Their task, in-context denoising, hands the model a set of stored patterns plus a corrupted query. The job is to recover the clean original. Dense associative memories, modern Hopfield networks, solve such tasks by sliding downhill on an energy landscape.

Working from probability theory, the team proves restricted versions of the task are solved perfectly by one transformer layer. A trained layer, they show, performs exactly one downhill step on a memory landscape built from the context, with the query as the starting point. That single step can beat simply snapping to the nearest stored pattern. The result tightens a known link between attention and memory, and gives in-context learning a concrete mechanical picture.

Catalog record

CREATORMatthew Smart, Alberto Bietti, Anirvan M. Sengupta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-05164

Browse the full library · About the Tower of Babel