WuNeng: Hybrid State with Attention
Liu Xiao, Li Zhiyuan, Lin Yueyu · 2025
Read WuNeng: Hybrid State with Attention by Liu Xiao, Li Zhiyuan, Lin Yueyu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19191.txt.
Description
A team of authors presents a language model design that blends two styles of sequence handling. Their 2025 paper joins standard attention heads with heads driven by a running state from a recurrent design. The aim is richer grasp of long context, rather than a smaller memory store.
Extra state heads sit beside the standard ones instead of taking their place. Heads trade signals through mixing, shifting, and gating steps, and the running state is read across many tokens at once. Those moves add few new parameters, the authors report. Early claims pitch stronger reasoning and text output at low extra cost. The design is offered as a fresh balance point between power and efficiency.
Catalog record
| CREATOR | Liu Xiao, Li Zhiyuan, Lin Yueyu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19191