Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention

Athanasios Zeris iD 0009-0002-6907-2400 · 2026

Read Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention by Athanasios Zeris iD 0009-0002-6907-2400 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26355.txt.

Description

Athanasios Zeris's 2026 paper adds two biases that standard transformer attention lacks: energy salience, which tokens gather information energy, and scale-wise reach, how far influence extends at each frequency. Energy-Gated Attention gates value pooling by a learned energy estimate from a single linear mapping, choosing what to attend to. Morlet Positional Encoding swaps fixed sine codes for learned Gaussian-window wavelets, deciding where focus works at each scale. Alone, each part helps a little, but combined they beat the sum of their parts, the paper's central finding.

Catalog record

CREATORAthanasios Zeris iD 0009-0002-6907-2400
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28825
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26355

Browse the full library · About the Tower of Babel