Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
Athanasios Zeris iD 0009-0002-6907-2400 · 2026
Read Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention by Athanasios Zeris iD 0009-0002-6907-2400 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26355.txt.
Description
Athanasios Zeris's 2026 paper adds two biases that standard transformer attention lacks: energy salience, which tokens gather information energy, and scale-wise reach, how far influence extends at each frequency. Energy-Gated Attention gates value pooling by a learned energy estimate from a single linear mapping, choosing what to attend to. Morlet Positional Encoding swaps fixed sine codes for learned Gaussian-window wavelets, deciding where focus works at each scale. Alone, each part helps a little, but combined they beat the sum of their parts, the paper's central finding.
Catalog record
| CREATOR | Athanasios Zeris iD 0009-0002-6907-2400 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28825 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26355