Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories

Sidi Lu, Zhenwen Liang, Dongyang Ma, Yan Wang, Haitao Mi, Dong Yu · 2026

Read Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories by Sidi Lu, Zhenwen Liang, Dongyang Ma, Yan Wang, Haitao Mi, Dong Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05085.txt.

Description

Long conversations and whole books overflow a language model's context window. Sidi Lu and 5 co-authors present Locas in this 2026 technical report from Tencent AI Lab, a parametric memory that stores context inside the model's own weights. The memory block mimics the feed-forward layers of a transformer, so it can be offloaded, merged in, or trained on its own.

Two variants are offered: a plain two-layer design with clearer theory, and one matching the gated feed-forward structure of modern large models for easy attachment. The key finding is that initialization matters most: seeding the memory from the model's own parameters, activations, or gradients gives fast convergence, better generalization, and protection against forgetting. On a whole-book task and a long-dialogue benchmark, Locas stored past context with as little as 0.02 percent extra parameters, and general ability survived largely intact.

Catalog record

CREATORSidi Lu, Zhenwen Liang, Dongyang Ma, Yan Wang, Haitao Mi, Dong Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-05085

Browse the full library · About the Tower of Babel