Do Value Vectors in Deep Layers Need Context from the Residual Stream?
Muyu He * 1 Yuchen Liu * 1 Qingya Huang 1 Li Zhang 2 1 Independent 2 Drexel University * Equal contribution. † Corresponding author. · 2026
Read Do Value Vectors in Deep Layers Need Context from the Residual Stream? by Muyu He * 1 Yuchen Liu * 1 Qingya Huang 1 Li Zhang 2 1 Independent 2 Drexel University * Equal contribution. † Corresponding author. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02780.txt.
Description
He and co-authors ask whether deep attention layers need context to build their value vectors. Attention layers normally read the residual stream. Query, key, and value vectors all become context-dependent. The team finds a surprise. Performance improves when deeper layers learn a context-free value vector. That vector preserves the original token information. Adding the context-dependent part back gives little extra benefit on aggregate benchmarks.
Context-free values can be stored as sparse parameters. No recompute or persistent cache is needed for them. Ablations map the key design choices. The result is Bank of Values, a lookup table of token-specific value vectors. It covers the last third of layers. On 135M and 780M models, BoV improves validation loss over standard attention. At 780M, it matches the previous best method's average across 21 benchmarks. It does so with less compute and memory.
Catalog record
| CREATOR | Muyu He * 1 Yuchen Liu * 1 Qingya Huang 1 Li Zhang 2 1 Independent 2 Drexel University * Equal contribution. † Corresponding author. |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.02780