Transformers with Selective Access to Early Representations
Skye Gunasekaran, Téa Wright, Rui-Jie Zhu, Jason Eshraghian · 2026
Read Transformers with Selective Access to Early Representations by Skye Gunasekaran, Téa Wright, Rui-Jie Zhu, Jason Eshraghian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03953.txt.
Description
Skye Gunasekaran, Téa Wright, Rui-Jie Zhu, and Jason Eshraghian treat access to early transformer representations as a retrieval problem. Several recent architectures let later layers see first-layer values, because low-level features can become hard to recover after repeated transformations. The cheapest versions use static value residuals that expose those values uniformly across tokens and heads, ignoring context.
Their Selective Access Transformer keeps the first-layer value pathway but controls it with a context-dependent gate, so each position takes only as much early information as it needs. Across models from 130 million to 1.3 billion parameters, the gated design consistently improved validation loss and zero-shot accuracy over static value-residual and plain transformer baselines, with the strongest gains on retrieval-intensive benchmarks.
Catalog record
| CREATOR | Skye Gunasekaran, Téa Wright, Rui-Jie Zhu, Jason Eshraghian |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.03953