Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences

Neehal Tumma, Noel Loo, Daniela Rus · 2026

Read Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences by Neehal Tumma, Noel Loo, Daniela Rus (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21100.txt.

Description

Neehal Tumma, Noel Loo, and Daniela Rus improve recurrent sequence models by accounting for curvature, the shape of the loss surface that delta-rule updates ignore. Starting from online least-squares theory, they show exact preconditioning links linear attention to the delta rule, then build a cheap diagonal version. Preconditioned DeltaNet, Gated DeltaNet, and Kimi Delta Attention variants gain steadily on synthetic recall tests and language modeling at 340M and 1B scales.

Catalog record

CREATORNeehal Tumma, Noel Loo, Daniela Rus
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE151225
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21100

Browse the full library · About the Tower of Babel