CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting
Viresh Pati, Yubin Kim, Vinh Pham, Jevon Twitty, Shihao Yang, Jiecheng Lu · 2026
Read CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting by Viresh Pati, Yubin Kim, Vinh Pham, Jevon Twitty, Shihao Yang, Jiecheng Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02729.txt.
Description
Viresh Pati and 5 co-authors redesign attention for time series forecasting so that trend, shock, and season stop tangling inside one softmax, in a 2026 paper. Standard attention mixes those three structures through global normalization. Recurrent models restore order dependence but give up long-range, order-free selection.
Their mechanism, CAPS, pairs rotation-based phase alignment with three additive gating paths: a Riemann softmax, prefix-product gates, and a learned clock baseline. The clock supplies a shared notion of time-based importance that steers all paths jointly. All fits inside a single attention layer with linear complexity. Tests on long- and short-term forecasting benchmarks beat vanilla softmax and linear attention. Results stayed competitive against seven strong rivals. The authors release their code publicly, casting structured attention as the fix for forecasting's mixed timescales.
Catalog record
| CREATOR | Viresh Pati, Yubin Kim, Vinh Pham, Jevon Twitty, Shihao Yang, Jiecheng Lu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02729