Transformers with Selective Access to Early Representations

Skye Gunasekaran, Téa Wright, Rui-Jie Zhu, Jason Eshraghian · 2026

Read Transformers with Selective Access to Early Representations by Skye Gunasekaran, Téa Wright, Rui-Jie Zhu, Jason Eshraghian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03953.txt.

Description

Skye Gunasekaran, Téa Wright, Rui-Jie Zhu, and Jason Eshraghian treat access to early transformer representations as a retrieval problem. Several recent architectures let later layers see first-layer values, because low-level features can become hard to recover after repeated transformations. The cheapest versions use static value residuals that expose those values uniformly across tokens and heads, ignoring context.

Their Selective Access Transformer keeps the first-layer value pathway but controls it with a context-dependent gate, so each position takes only as much early information as it needs. Across models from 130 million to 1.3 billion parameters, the gated design consistently improved validation loss and zero-shot accuracy over static value-residual and plain transformer baselines, with the strongest gains on retrieval-intensive benchmarks.

Catalog record

CREATORSkye Gunasekaran, Téa Wright, Rui-Jie Zhu, Jason Eshraghian
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03953

Browse the full library · About the Tower of Babel