Gated Delta Networks: Improving Mamba2 with Delta Rule

Songlin Yang, Jan Kautz, Ali Hatamizadeh ⋆ · 2024

Read Gated Delta Networks: Improving Mamba2 with Delta Rule by Songlin Yang, Jan Kautz, Ali Hatamizadeh ⋆ (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06464.txt.

Description

Songlin Yang, Jan Kautz, and Ali Hatamizadeh merge two memory tricks to make sequence models both forget fast and edit precisely, in a paper published in two thousand twenty-four. Linear transformers promise cheaper long-context modeling than standard attention, but they lag on retrieval and long documents.

Two recent mechanisms attack different halves of the problem. Gating lets the model erase memory quickly when content turns stale. The delta rule makes surgical updates to what is stored. The authors observe the two are complementary, so they fuse them into a gated delta rule. A parallel training algorithm makes the fused rule fast on modern hardware. The resulting architecture, Gated DeltaNet, beats Mamba2 and DeltaNet across language modeling, reasoning, retrieval, and long-context benchmarks. Hybrid versions that mix in ordinary attention layers push performance further.

Catalog record

CREATORSonglin Yang, Jan Kautz, Ali Hatamizadeh ⋆
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06464

Browse the full library · About the Tower of Babel