Length Generalization with Log-Depth Recurrent Units

Charles Pert [email protected], Dalal Alrajeh [email protected], Alessandra Russo [email protected] · 2026

Read Length Generalization with Log-Depth Recurrent Units by Charles Pert [email protected], Dalal Alrajeh [email protected], Alessandra Russo [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26035.txt.

Description

Pert, Alrajeh and Russo tackle length generalisation, where models trained on short sequences fail on longer ones. Recurrent models carry positional biases, while transformers have fixed compute depth. The authors propose MLP-LDRU, a log-depth recurrent unit built from associativity-biased operators that mimic recurrence by parallel reduction. On 21 regular-language tasks, it hits 100 percent out-of-distribution accuracy on 18 and at least 99.9 percent on the rest as training length grows. It also competes beyond regular languages on classification-style benchmarks.

Catalog record

CREATORCharles Pert [email protected], Dalal Alrajeh [email protected], Alessandra Russo [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE123821
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26035

Browse the full library · About the Tower of Babel