Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories Sidi Lu, Zhenwen Liang, Dongyang Ma, Yan Wang, Haitao Mi, Dong Yu Abstract In this paper, we aim to bridge test-time-training with a new type of parametric memory that can be flexibly offloaded from or merged into model parameters. We present Locas, a Locally-Supported parametric memory that shares the design of FFN blocks in modern transformers, allowing it to be flexibly permanentized into the model parameters while supporting efficient continual learning. We discuss two major variants of Locas: one with a conventional two-layer MLP design that has a clearer theoretical guarantee; the other one shares the same GLU-FFN structure with SOTA LLMs, and can be easily attached to existing models for both parameter-efficient and computation-efficient continual learning. Crucially, we show that proper initialization of such low-rank sideway-FFN-style memories—performed in a principled way by reusing model parameters, activations and/or gradients—is essential for fast convergence, improved generalization, and catastrophic forgetting prevention. We validate the proposed memory mechanism on the PG-19 whole-book language modeling and LoCoMo long-context dialogue question answering tasks. With only 0.02% additional parameters in the lowest case, Locas-GLU is capable of storing the information from past context while maintaining a much smaller context window. In addition, we also test the model’s general capability loss after memorizing the whole book with Locas, through comparative MMLU evaluation. Results show the promising ability of Locas to permanentize past context into parametric knowledge with minimized catastrophic forgetting of the model’s existing internal knowledge. ]Tencent AI Lab \contribution Correspondence to: sidilu@global.tencent.com 1 Introduction Test-time adaptation of language models has long been a central problem for reliable deployment. Generally, existing approaches fall under two paradigms: non-parametric and parametric mechanisms. The dominant non-parametric paradigm is in-context learning (ICL) [ 2 ] : the model conditions on prompts that contain demonstrations, tool outputs [ 26 ] and additional information from retrieval-based augmentation (RAG) [ 17 ] , then produces task-appropriate behavior without updating weights. This approach is appealing, as it is stable and deployment-friendly, yet its “learning” is mediated entirely through attention over the prompt. As a result, the adaptation is bounded by context length [ 18 ] and can be overly restricted to certain formatting, ordering and distractors [ 39 , 28 ] , which can manifest as controllability failures, including susceptibility to prompt injection and related jailbreak-style attacks [ 20 , 40 ] . In contrast, test-time training (TTT) methods [ 32 ] perform online parametric updates using self-supervised signals available at inference. While these techniques can in principle internalize new information beyond the prompt, they often incur additional optimization cost (typically multiple gradient-based iterations per token step) [ 35 ] , and they require careful objective design to avoid catastrophic distribution shift while still enabling useful adaptivity. In this paper, we try to answer the following question: Can we significantly improve the parameter and compute efficiency of test-time training through principled initialization of the memory module? We argue that proper initialization can dramatically accelerate convergence and reduce the number of parameters required for effective memorization [ 7 ] . We propose Locas, a locally-supported parametric memory that achieves both parameter and compute efficiency through principled initialization. In contrast to previous TTT methods that operate through applying gradient to all or part of the model parameters where the introduced parts adopt random initialization [ 35 ] , Locas adaptively uses the backbone model’s behavior to guide the initialization of the newly introduced memory module. We discuss two general variants of Locas, including the more theoretically sound one called Locas-MLP with a two-layer MLP structure, and the more practically flexible one called Locas-GLU that shares the GLU-FFN structure with SOTA LLMs [ 27 ] . To maintain a compact parametric memory with minimized information loss, we also develop an efficient compression algorithm for such memory. We generalize the traditional SVD algorithm [ 6 ] , which only works on linear projections, to the two-layer non-linear case, and propose the Non-Linear SVD (NL-SVD) algorithm to efficiently compute the best compressed form with lower latent dimension. Contributions. • We propose Locas, a Locally-supported parametric memory that utilizes the backbone model it relies on to perform principled initialization, achieving boosted computation and parameter efficiency, fast convergence and improved generalization. • We discuss the expansion and compression of such parametric memory w.r.t. the growth of longer context. Specifically: – For Locas-MLP, we show that the proposed initialization is step-wise optimal (both time-step and gradient-update-step) under certain assumptions; for Locas-GLU, we show that activation-guided parameter cloning anchors the memory initialization within the principal subspace of the original FFN parameters. – We propose the Non-Linear SVD for compression of the Locas-MLP variant (see Appendix A.1 ), while noting its practical limitations compared to standard backpropagation. • We evaluate the proposed Locas against existing methods including the full attention baseline, context-truncation baseline and the TempLoRA baseline on PG-19 whole-book language modeling and Locomo long-context dialogue QA tasks [ 35 , 24 , 21 ] , demonstrating significant parameter and compute efficiency advantages. 2 Methodology We hereby introduce Locas, a locally-supported parametric memory framework that achieves parameter and compute efficiency through principled initialization. We first revisit the Feed-Forward Network (FFN) in transformers, and reinterpret it as a soft-lookup table or internal attention mechanism. Figure 1 : Illustration of a typical dense transformer layer with FFN interpreted as a soft look-up table memory, in comparison with the attention mechanism, which is a contextual soft look-up table mechanism. The GLU variant follows a similar structure but with an additional gating mechanism. 2.1 Notation: FFNs are Soft Look-up Table Memories We begin by introducing notation and making explicit the interpretation of feed-forward networks (FFNs) as parametric memories. Consider a classical transformer layer indexed by i , with hidden dimensionality d and FFN intermediate dimensionality m . Let \mathcal{A}_{t}^{i}\in\mathbb{R}^{d} denote the input activation to the FFN at token position t and layer i , after the attention sublayer and layer normalization. A classical two-layer FFN with element-wise nonlinearity \phi(\cdot) computes \mathrm{FFN}(\mathcal{A}_{t}^{i})=V^{\top}\,\phi\!\left(K^{\top}\mathcal{A}_{t}^{i}\right), (1) where K\in\mathbb{R}^{d\times m} is the first-layer weight matrix (which we call "key matrix" later on) and V\in\mathbb{R}^{m\times d} is the second-layer weight matrix (which we call "value matrix" later on) . We omit bias terms for clarity, as they do not affect the arguments that follow. Let k_{j}\in\mathbb{R}^{d} denote the j -th column of K and v_{j}\in\mathbb{R}^{d} denote the j -th row of V . The FFN output can then be written as a sum over intermediate dimensions: \mathrm{FFN}(\mathcal{A}_{t}^{i})=\sum_{j=1}^{m}\phi\!\left(\langle\mathcal{A}_{t}^{i},k_{j}\rangle\right)\,v_{j}. (2) This decomposition admits a direct interpretation as a soft look-up table or internal attention mechanism. Each pair (k_{j},v_{j}) defines a memory slot: the key vector k_{j} determines when the slot is activated via its inner product with the input, while the value vector v_{j} determines what content is retrieved. The nonlinearity \phi(\cdot) acts as an activation gate that sparsifies or modulates the contribution of each memory slot. Under this view, the FFN implements an unnormalized linear attention mechanism [ 16 ] with a fixed, parametric set of m key–value pairs: \mathrm{FFN}(\mathcal{A}_{t}^{i})=\sum_{j=1}^{m}\alpha_{j}(\mathcal{A}_{t}^{i})\,v_{j},\qquad\alpha_{j}(\mathcal{A}_{t}^{i})\doteq\phi\!\left(\langle\mathcal{A}_{t}^{i},k_{j}\rangle\right). (3) Unlike self-attention, these keys and values are not derived from the input sequence but are instead stored directly in the model parameters. As a result, the FFN can be viewed as a persistent, content-addressable memory whose capacity is proportional to its intermediate dimensionality m . This reinterpretation highlights a critical observation: increasing m is equivalent to increasing the number of memory slots available to the model, while modifying (k_{j},v_{j}) corresponds to writing new entries into this parametric memory. In modern large transformers, FFNs typically dominate parameter count, suggesting that they are the primary locus of memorization capacity. In the remainder of this section, we exploit this perspective to construct an explicit mechanism for writing new key–value pairs into FFNs at test time with principled initialization, thereby enabling efficient parametric memory expansion with fast convergence. Figure 2 shows how the proposed parametric memory works in transformer layers. 2.2 Two Variants: Locas-MLP and Locas-GLU Before discussing the initialization strategies, we first introduce two variants of Locas that cater to different model architectures and use cases. Locas-MLP This variant adopts a conventional two-layer MLP structure with ReLU activation: \text{Locas-MLP}(\mathcal{A})=V^{\top}\cdot\text{ReLU}(K^{\top}\mathcal{A}) where K\in\mathbb{R}^{d\times r} is the key matrix and V\in\mathbb{R}^{r\times d} is the value matrix, with r being the latent dimensionality of the memory. This variant has clearer theoretical guarantees as both its key and value matrices admit step-wise optimal closed-form solutions. However, it has compatibility issues with models that were not trained with two-layer MLP-style FFN blocks - specifically, the piecewise-linear separability of representations at the MLP input is often poor for such models. Locas-GLU This variant shares the GLU-FFN structure with state-of-the-art LLMs (e.g., LLaMA, Qwen, Mistral): \text{Locas-GLU}(\mathcal{A})=V^{\top}\cdot(\sigma(G^{\top}\mathcal{A})\odot K^{\top}\mathcal{A}) where G\in\mathbb{R}^{d\times r} is the gate matrix, K\in\mathbb{R}^{d\times r} is the up-projection (key) matrix, V\in\mathbb{R}^{r\times d} is the down-projection (value) matrix, and \sigma(\cdot) is the SiLU activation function. This variant can be seamlessly attached to existing GLU-based models for both parameter-efficient and computation-efficient continual learning. Figure 2 : Architecture of the proposed Locas parametric memory integrated as a sideway FFN module in transformer layers. The memory module operates in parallel with the backbone FFN, with its output scaled and added to the main pathway. This design enables genuine model capacity expansion at test time while preserving the backbone model’s pretrained representations. 2.3 Memory Initialization for Locas-MLP: Activation and Gradient Reusage Yields Step-wise Optimal Initialization Given the memorized token x_{t} with context x_{