Length Generalization with Log-Depth Recurrent Units
Charles Pert [email protected], Dalal Alrajeh [email protected], Alessandra Russo [email protected] · 2026
Read Length Generalization with Log-Depth Recurrent Units by Charles Pert [email protected], Dalal Alrajeh [email protected], Alessandra Russo [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26035.txt.
Description
Pert, Alrajeh and Russo tackle length generalisation, where models trained on short sequences fail on longer ones. Recurrent models carry positional biases, while transformers have fixed compute depth. The authors propose MLP-LDRU, a log-depth recurrent unit built from associativity-biased operators that mimic recurrence by parallel reduction. On 21 regular-language tasks, it hits 100 percent out-of-distribution accuracy on 18 and at least 99.9 percent on the rest as training length grows. It also competes beyond regular languages on classification-style benchmarks.
Catalog record
| CREATOR | Charles Pert [email protected], Dalal Alrajeh [email protected], Alessandra Russo [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 123821 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26035