A Random Matrix Analysis of In-context Memorization for Nonlinear Attention

Zhenyu Liao, Jiaqing Liu, TianQi Hou, Difan Zou, Zenan Ling · 2025

Read A Random Matrix Analysis of In-context Memorization for Nonlinear Attention by Zhenyu Liao, Jiaqing Liu, TianQi Hou, Difan Zou, Zenan Ling (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18656.txt.

Description

Zhenyu Liao, Jiaqing Liu, TianQi Hou, Difan Zou, and Zenan Ling wrote this paper in two thousand twenty-five. It measures, with random matrix tools, how much error nonlinear attention makes when it memorizes data in context.

Attention is the core block of modern models, yet its theory lags its use, most of all in the nonlinear case. The team works where token counts and widths are both huge and alike in size. For fixed attention weights on signal-plus-noise data, exact limiting forms for the mean squared error are derived. On pure noise, attention errs more than plain linear regression. When the input holds a clean signal and the weights line up with it, the gap closes and can flip. Tests on numbers back the math.

Catalog record

CREATORZhenyu Liao, Jiaqing Liu, TianQi Hou, Difan Zou, Zenan Ling
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE165 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-18656

Browse the full library · About the Tower of Babel