CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting

Viresh Pati, Yubin Kim, Vinh Pham, Jevon Twitty, Shihao Yang, Jiecheng Lu · 2026

Read CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting by Viresh Pati, Yubin Kim, Vinh Pham, Jevon Twitty, Shihao Yang, Jiecheng Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02729.txt.

Description

Viresh Pati and 5 co-authors redesign attention for time series forecasting so that trend, shock, and season stop tangling inside one softmax, in a 2026 paper. Standard attention mixes those three structures through global normalization. Recurrent models restore order dependence but give up long-range, order-free selection.

Their mechanism, CAPS, pairs rotation-based phase alignment with three additive gating paths: a Riemann softmax, prefix-product gates, and a learned clock baseline. The clock supplies a shared notion of time-based importance that steers all paths jointly. All fits inside a single attention layer with linear complexity. Tests on long- and short-term forecasting benchmarks beat vanilla softmax and linear attention. Results stayed competitive against seven strong rivals. The authors release their code publicly, casting structured attention as the fix for forecasting's mixed timescales.

Catalog record

CREATORViresh Pati, Yubin Kim, Vinh Pham, Jevon Twitty, Shihao Yang, Jiecheng Lu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02729

Browse the full library · About the Tower of Babel