Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories
Sidi Lu, Zhenwen Liang, Dongyang Ma, Yan Wang, Haitao Mi, Dong Yu · 2026
Read Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories by Sidi Lu, Zhenwen Liang, Dongyang Ma, Yan Wang, Haitao Mi, Dong Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05085.txt.
Description
Long conversations and whole books overflow a language model's context window. Sidi Lu and 5 co-authors present Locas in this 2026 technical report from Tencent AI Lab, a parametric memory that stores context inside the model's own weights. The memory block mimics the feed-forward layers of a transformer, so it can be offloaded, merged in, or trained on its own.
Two variants are offered: a plain two-layer design with clearer theory, and one matching the gated feed-forward structure of modern large models for easy attachment. The key finding is that initialization matters most: seeding the memory from the model's own parameters, activations, or gradients gives fast convergence, better generalization, and protection against forgetting. On a whole-book task and a long-dialogue benchmark, Locas stored past context with as little as 0.02 percent extra parameters, and general ability survived largely intact.
Catalog record
| CREATOR | Sidi Lu, Zhenwen Liang, Dongyang Ma, Yan Wang, Haitao Mi, Dong Yu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-05085