HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing

Yuan Fang, Xuming Ran, [0.3em] IXL Learning, Inc · 2026

Read HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing by Yuan Fang, Xuming Ran, [0.3em] IXL Learning, Inc (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08143.txt.

Introduction

Yuan Fang, Yi Xie, and Xuming Ran improve lifelong editing of large language models, where single facts are patched after deployment without retraining. Existing editors fade as edits pile up and often fail when a question is reworded, so a fix that works once may not stick.

HoReN wraps one MLP layer with a discrete codebook memory that leaves the model's weights untouched. Each entry acts as both a knowledge key and a Hopfield pattern, and edits are retrieved by angle on a sphere, then refined by damped Hopfield dynamics. Paraphrases settle into the right memory basin while unrelated inputs stay stable, and results hold across editing benchmarks.

Catalog record

CREATORYuan Fang, Xuming Ran, [0.3em] IXL Learning, Inc
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08143

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing Yuan Fang, Xuming Ran, [0.3em] IXL Learning, Inc Abstract Large language models encode vast factual knowledge that can become outdated or incorrect after deployment, yet retraining is prohibitively costly. This motivates lifelong model editing, which updates targeted behavior while preserving the rest of the model. Existing editors, both parameter-modifying and parameter-preserving, degrade severely as edits accumulate and struggle to generalize across paraphrases. We propose HoReN , a codebook-based parameter-preserving editor that wraps a single MLP layer with a discrete key–value memory. HoReN treats each codebook entry as both a knowledge key and a Hopfield stored pattern, retrieves edits by angular similarity on the unit hypersphere, and refines queries through damped Hopfield dynamics so paraphrases converge to the correct memory basin while unrelated inputs remain stable. HoReN achieves strong editing performance with consistent gains across diverse benchmarks spanning standard ZsRE, structured WikiBigEdit, and unstructured UnKE evaluations. Moreover, HoReN scales to 50K sequential edits on ZsRE with stable overall performance above 0.93, while prior editors collapse or degrade severely before reaching 10K. Our code is available at https://github.com/ha11ucin8/HoReN . 1 Introduction Large language models (LLMs) [ 3 , 29 , 24 , 1 ] are routinely deployed against a world whose facts drift, whose deployments surface hallucinations, and whose operators need to inject corrections that were absent at pre-training time. Re-training end-to-end for every update is prohibitively expensive and risks catastrophic forgetting [ 10 , 23 , 28 , 16 ] , motivating model editing : targeted procedures that revise an LLM’s behavior on a specified prompt while leaving its other behaviors intact, with edits arriving as a continual stream [ 17 , 6 , 26 , 27 , 15 ] . Existing editors fall into two paradigms: parameter-modifying methods alter base weights, including the locate-and-edit line (ROME [ 17 ] , MEMIT [ 18 ] , AlphaEdit [ 4 ] ) and one-shot direct shifts (UltraEdit [ 5 ] ); parameter-preserving methods leave the base weights frozen and route edits through external memory at a chosen layer, as in GRACE’s discrete codebook [ 6 ] and WISE’s gated side-memory [ 26 ] . Figure 1 : Overall Performance (OP, geometric mean of Reliability, Generalization, and Locality) scaling from 100 to 50\text{K} sequential edits on ZsRE (LLaMA-3.1-8B). HoReN sustains OP \geq 0.93 through 50K edits, while every baseline collapses or plateaus before 10K: ROME fails immediately ( 0.03 ), GRACE flatlines at 0.37 – 0.38 , WISE drifts down to 0.51 , AlphaEdit cliffs from 0.82 at 2K to 0.08 at 10K, and UltraEdit drifts down to 0.67 . Beyond 10K only HoReN remains within the 0.90 stability band. Per-metric breakdowns and the per-checkpoint table are in Appendix F.1 . Parameter-modifying editors have shown notable success on small batches of edits, but directly perturbing weights trained on trillions of tokens disrupts preserved knowledge as edits accumulate, and constraint-based remedies such as AlphaEdit’s null-space projection [ 4 ] only partially mitigate this drift in a sequential regime (Figure 1 ). Parameter-preserving editors take a different route. GRACE [ 6 ] writes new mappings into a chosen layer’s latent space as a discrete key–value codebook of adaptors and routes by a hard-radius nearest-neighbor lookup that defers to a stored value whenever an incoming activation falls within the radius of a stored key. WISE [ 26 ] maintains a dual parametric memory, a frozen main memory of pretrained weights and a trainable side memory holding edits with a router selecting between them, and uses a knowledge-sharding mechanism that places distinct edit subsets in disjoint subspaces before merging them into the side memory. Both designs protect existing knowledge by construction, yet their generalization and overall performance degrade as the edit stream scales (Figure 1 ). The failure is fundamentally a routing problem: the lookup or router cannot reliably map a paraphrased query to the correct stored edit. To address these routing failures in parameter-preserving editors, we propose HoReN ( Ho pfield Re trieval with N ormalized representations), which combines three components. (i) The codebook architecture of GRACE [ 6 ] : a frozen base model with one MLP layer wrapped by a discrete key–value memory, so each edit is stored once and never re-touched. (ii) Normalization : keys and queries are projected onto the unit hypersphere so retrieval depends on angular similarity rather than activation magnitude. (iii) Hopfield dynamics [ 22 , 11 ] : the codebook is read as a field of energy basins, and one damped attractor step before matching lets a paraphrase relax into the basin of the correct memory while leaving unrelated queries essentially undisturbed. The matched edit is realized by a lightweight value adaptor at the same layer l , so the entire intervention is concentrated on the routing decision. The two novel components play complementary roles: normalization protects locality and enables generalization by keeping unrelated queries separated on the hypersphere, while one damped Hopfield step closes the generalization gap by pulling rephrased queries into the basin of the correct edit. The result is a sequential editor that scales without catastrophic degradation: on a controlled ZsRE stress test on LLaMA-3.1-8B, HoReN keeps reliability, generalization, and locality all above 89\% out to 50K accumulated edits (Figure 1 ), where every baseline has cliffed (AlphaEdit, ROME), drifted (UltraEdit, WISE), or flat-lined on generalization (GRACE) well before 10K. The same advantage carries over to structured WikiBigEdit [ 25 ] and unstructured UnKE [ 2 ] . Related works are shown in Appendix B . Our main contributions are: • We characterize a structural divide in lifelong editing: parameter-modifying editors degrade preserved knowledge even with constraint measures, while parameter-preserving editors lose generalization because magnitude-sensitive nearest-neighbor routing cannot bridge the representation gap between an edit prompt and its rephrasings. • We propose HoReN, a parameter-preserving editor that unifies the ROME key–value memory view with the GRACE codebook architecture under a normalized Hopfield retrieval rule, and provide theoretical analysis showing that iterating Hopfield dynamics to convergence attracts all queries, including unrelated ones, toward stored codes, formally motivating HoReN’s single-step deployment. • We demonstrate large-scale stability of HoReN on a 50K-edit ZsRE stress test on LLaMA-3.1-8B, where prior methods cliff, drift, or flat-line well before 10K edits, and confirm consistent gains on structured WikiBigEdit and unstructured UnKE across seven LLMs spanning four model families. Figure 2 : Overall architecture of HoReN combining normalized representations with Hopfield-style retrieval. The pipeline: (1) extract hidden states from layer l ; (2) pool token representations to construct the query; (3) normalize query to the unit hypersphere; (4) apply one-step Hopfield-style refinement over the normalized codebook; (5) perform key matching and value adaptation at layer l . This design achieves complementary benefits: normalization preserves locality through angular similarity matching, while Hopfield-style refinement improves paraphrase routing by moving the query toward edit-key attractors. 2 Method HoReN inherits the architectural skeleton of GRACE [ 6 ] : the base model f_{\theta_{0}} is frozen and a single MLP layer l is wrapped with a discrete codebook \mathcal{C}=\{(k_{i},v_{i},y_{i})\}_{i=1}^{C} (Figure 2 ). At inference, an incoming query is routed through the codebook; a successful match modifies the layer’s output by adding the matched value to the output, while an

Read the complete text