A Random Matrix Analysis of In-context Memorization for Nonlinear Attention
Zhenyu Liao, Jiaqing Liu, TianQi Hou, Difan Zou, Zenan Ling · 2025
Read A Random Matrix Analysis of In-context Memorization for Nonlinear Attention by Zhenyu Liao, Jiaqing Liu, TianQi Hou, Difan Zou, Zenan Ling (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18656.txt.
Description
Zhenyu Liao, Jiaqing Liu, TianQi Hou, Difan Zou, and Zenan Ling wrote this paper in two thousand twenty-five. It measures, with random matrix tools, how much error nonlinear attention makes when it memorizes data in context.
Attention is the core block of modern models, yet its theory lags its use, most of all in the nonlinear case. The team works where token counts and widths are both huge and alike in size. For fixed attention weights on signal-plus-noise data, exact limiting forms for the mean squared error are derived. On pure noise, attention errs more than plain linear regression. When the input holds a clean signal and the weights line up with it, the gap closes and can flip. Tests on numbers back the math.
Catalog record
| CREATOR | Zhenyu Liao, Jiaqing Liu, TianQi Hou, Difan Zou, Zenan Ling |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 165 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18656