Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse

Chi Zhang, Mengqi Zhang, Xiaotian Ye, Runxi Cheng, Zisheng Zhou, Ying Zhou, Pengjie Ren, Zhumin Chen · 2026

Read Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse by Chi Zhang, Mengqi Zhang, Xiaotian Ye, Runxi Cheng, Zisheng Zhou, Ying Zhou, Pengjie Ren, Zhumin Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11042.txt.

Introduction

Chi Zhang and co-authors study the collapse that hits large language models under sequential knowledge editing. Editing updates a model's facts one after another. After many edits, the model's general abilities can fall apart. Parameter-modifying edits are the riskiest kind. Past fixes lean on heuristic limits to how far parameters may move.

The authors characterize the collapse through spectral signatures, patterns in the model's weight matrices. A spectral view may show why edits pile up damage. It could also yield an early warning and a cleaner fix than heuristics. The work serves teams that must keep deployed models current without slowly breaking them.

Catalog record

CREATORChi Zhang, Mengqi Zhang, Xiaotian Ye, Runxi Cheng, Zisheng Zhou, Ying Zhou, Pengjie Ren, Zhumin Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11042

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse Chi Zhang, Mengqi Zhang, Xiaotian Ye, Runxi Cheng, Zisheng Zhou, Ying Zhou, Pengjie Ren, Zhumin Chen Abstract Sequential knowledge editing in large language models often causes catastrophic collapse of the model’s general abilities, especially for parameter-modifying methods. Existing approaches mitigate this issue through heuristic constraints on parameter updates, yet the mechanisms underlying such degradation remain insufficiently understood. In this work, we present a spectral analysis of sequential knowledge editing and show that a model’s general abilities are closely associated with dominant singular directions of pretrained weight matrices. These directions are highly sensitive to perturbations and are progressively disrupted by repeated edits, closely tracking the collapse in both editing efficacy and general performance. Building on this insight, we propose REVIVE, a plug-and-play framework that stabilizes sequential editing by explicitly preserving the dominant singular subspace. REVIVE represents parameter updates in the spectral basis of the original weights and filters components that would interfere with the protected region. Extensive experiments across multiple models and benchmarks show that REVIVE consistently improves editing efficacy while substantially preserving general abilities under long-horizon sequential editing, including extreme settings with up to 20,000 edits. 1 Introduction Large language models (LLMs) often generate outdated or incorrect information due to limitations in pretraining data or evolving real-world knowledge ( Cao et al. 2021 ) . Knowledge editing ( Meng et al. 2022b ) provides a lightweight way to correct specific facts without retraining models from scratch. In practice, however, edits are rarely isolated: models are updated repeatedly over time, leading to the setting of sequential knowledge editing ( Fang et al. 2024 ) , which requires both accurate edits and the preservation of general abilities ( Gu et al. 2024 ) . These challenges are particularly pronounced for parameter-modifying methods that directly update model weights, which are the focus of this work. Figure 1: Performance of representative sequential knowledge editing methods on CounterFact using LLaMA3. Despite the development of several sequential knowledge editing methods, including RECT ( Gu et al. 2024 ) , PRUNE ( Ma et al. 2024 ) , and AlphaEdit ( Fang et al. 2024 ) , long-horizon degradation remains largely unresolved. As illustrated in Figure 1 , although these approaches appear can effective in short or small-scale setting, they still suffer pronounced drops in both editing efficacy and general abilities as the number of edits increases. A key limitation is that most existing methods primarily mitigate interference across updates through heuristic constraints or regularization, such as bounding update magnitudes, reducing interference across edits, or minimizing disruption to unrelated knowledge, without explicitly modeling how parameter updates interact with the intrinsic structure of pretrained weights. In particular, they do not clarify which parts of the pretrained weights are most relevant to general abilities, or how repeated updates accumulate and gradually degrade them. As a result, these methods provide limited mechanistic insight into why performance progressively degrades as edits accumulate. In this work, we conduct a systematic analysis of sequential knowledge editing from a spectral perspective (§ 2 ). Through a detailed study of the singular value decomposition (SVD) of weight matrices, we obtain three key findings: (i) general abilities are concentrated in a low-rank spectral structure spanned by dominant singular directions (§ 2.2 ); (ii) this structure is unusually fragile, such that even small perturbations aligned with dominant directions can disproportionately degrade general performance (§ 2.3 ); and (iii) repeated edits progressively distort these dominant directions in practice (§ 2.4 ). Together, these findings provide strong empirical evidence that interference with the dominant singular subspace is closely tied to failure in sequential editing, suggesting a primary mechanism underlying this collapse. Despite this insight, preserving dominant spectral structure during editing is nontrivial: editing objectives are local and offer little direct control over global spectral properties, and parameter-modifying editors differ substantially across methods. To address these challenges, we develop a R obust s E quential editing V ia dom I nant subspace preser V ation fram E work (REVIVE) (§ 3 ), a plug-and-play framework that stabilizes sequential knowledge editing by explicitly protecting dominant spectral structure. The core idea of REVIVE is to analyze parameter updates in the singular vector basis of the original weight matrix, which enables a fine-grained characterization of how edits interact with intrinsic functional directions. Based on this representation, REVIVE identifies the dominant subspace using a spectral energy criterion and constructs constrained updates by filtering out components that interfere with this critical region. This design allows factual knowledge to be incorporated through lower-energy directions while preserving the dominant spectral structure associated with general abilities. Our contributions can be summarized as follows: • We present a systematic spectral analysis of sequential knowledge editing, revealing that interference with the dominant singular subspace of pretrained weight matrices is a key mechanism underlying model collapse. • Based on this analysis, we propose REVIVE, a simple and plug-and-play framework that stabilizes sequential knowledge editing by explicitly preserving the dominant singular subspace during parameter updates. • We conduct extensive experiments on multiple models and benchmarks, demonstrating that REVIVE consistently outperforms state-of-the-art methods in both editing efficacy and the preservation of general abilities under sequential editing. 2 Why Sequential Editing Collapses: A Spectral Perspective To understand why sequential editing leads to model collapse, this section presents a spectral analysis of parameter matrices. We begin by viewing each weight matrix as a composition of independent input-output mappings derived from its Singular Value Decomposition (SVD). Based on this perspective, we empirically investigate three key questions: (i) where a model’s general abilities are concentrated within these spectral components , (ii) how sensitive these components are to perturbations , and (iii) how sequential editing progressively distorts these components in practice . Together, these analyses lead to a unified insight: sequential editing fails when cumulative update progressively corrupt the dominant singular directions of weight matrices, which are strongly associated with general abilities. As mainstream editing methods primarily target feed-forward network (FFN) layers for modification ( Meng et al. 2022b ; Meng et al. 2022a ) , we ground our investigation in the FFN matrices of LLaMA3-8B. 2.1 Spectral View of Weight Matrices From a spectral perspective, a parameter matrix \mathbf{W}\in\mathbb{R}^{m\times n} can be decomposed into a set of independent input-output mappings using Singular Value Decomposition (SVD): \mathbf{W}=\mathbf{U}\Sigma\mathbf{V}^{\top}=\sum_{i=1}^{r}\sigma_{i}\mathbf{u}_{i}\mathbf{v}_{i}^{\top}, (1) where \mathbf{U}\in\mathbb{R}^{m\times m} and \mathbf{V}\in\mathbb{R}^{n\times n} contain orthogonal left and right singular vectors, respectively, and \Sigma\in\mathbb{R}^{m\times n} contains the singular values \sigma_{1}\geq\sigma_{2}\geq\dots\geq\sigma_{r}\geq 0 . Each rank-one component \sigma_{i}\mathbf{u}_{i}\mathbf{v}_{i}^{\top} acts as a distinct input-output mapping: an input vector \mathbf{x}\in\mathbb{R}^{n} is projected onto \mathbf{v}_{i} , scaled by \sigma_{i} , and expanded along \mathbf{u}_{i} to produce the output. The orthogonality of the singular vectors ensures these mappings operate independently. Pretraining learns this highly structured functional decomposition, making it a critical component of the model’s general abilities ( Wang et al. 2024b ; Hao et al. 2025 ) . Crucially, this spectral view provides a unified lens for analyzing the failure of sequential editing. In the remainder of this section, we use this perspective to systematically investigate the three questions outlined above. 2.2 Where Are General Abilities Concentrated? To identify where general abilities are concentrated, we evaluate model performance on GLUE tasks (MRPC, CoLA, RTE,

Read the complete text