Gated Delta Networks: Improving Mamba2 with Delta Rule
Songlin Yang, Jan Kautz, Ali Hatamizadeh ⋆ · 2024
Read Gated Delta Networks: Improving Mamba2 with Delta Rule by Songlin Yang, Jan Kautz, Ali Hatamizadeh ⋆ (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06464.txt.
Description
Songlin Yang, Jan Kautz, and Ali Hatamizadeh merge two memory tricks to make sequence models both forget fast and edit precisely, in a paper published in two thousand twenty-four. Linear transformers promise cheaper long-context modeling than standard attention, but they lag on retrieval and long documents.
Two recent mechanisms attack different halves of the problem. Gating lets the model erase memory quickly when content turns stale. The delta rule makes surgical updates to what is stored. The authors observe the two are complementary, so they fuse them into a gated delta rule. A parallel training algorithm makes the fused rule fast on modern hardware. The resulting architecture, Gated DeltaNet, beats Mamba2 and DeltaNet across language modeling, reasoning, retrieval, and long-context benchmarks. Hybrid versions that mix in ordinary attention layers push performance further.
Catalog record
| CREATOR | Songlin Yang, Jan Kautz, Ali Hatamizadeh ⋆ |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06464