Benchmarking and Rethinking Knowledge Editing for Large Language Models
Guoxiu He, Xin Song, Futing Wang, Aixin Sun · 2025
Read Benchmarking and Rethinking Knowledge Editing for Large Language Models by Guoxiu He, Xin Song, Futing Wang, Aixin Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18690.txt.
Introduction
Guoxiu He and co-authors benchmark ways to edit knowledge inside large language models in a 2025 paper. Editing should fix a fact without retraining the model. Methods split between changing weights and adding external memory, but inconsistent tests made them hard to compare.
Their study adds event-based and general datasets beyond single facts, tests instruction and reasoning models under realistic generation, and includes multi-edit rounds. A simple baseline using selective contextual reasoning is compared throughout. Sobering result: weight-editing methods fare poorly under realistic conditions. The benchmark resets expectations for a technique often oversold.
Catalog record
| CREATOR | Guoxiu He, Xin Song, Futing Wang, Aixin Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18690
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking and Rethinking Knowledge Editing for Large Language Models Guoxiu He, Xin Song, Futing Wang, Aixin Sun Abstract Knowledge editing aims to update the embedded knowledge within Large Language Models (LLMs). However, existing approaches, whether through parameter modification or external memory integration, often suffer from inconsistent evaluation objectives and experimental setups. To address this gap, we conduct a comprehensive benchmarking study. In addition to fact-level datasets, we introduce more complex event-based datasets and general-purpose datasets drawn from other tasks. Our evaluation covers both instruction-tuned and reasoning-oriented LLMs, under a realistic autoregressive inference setting rather than teacher-forced decoding. Beyond single-edit assessments, we also evaluate multi-edit scenarios to better reflect practical demands. We employ four evaluation dimensions, including portability, and compare all recent methods against a simple and straightforward baseline named Selective Contextual Reasoning (SCR). Empirical results reveal that parameter-based editing methods perform poorly under realistic conditions. In contrast, SCR consistently outperforms them across all settings. This study offers new insights into the limitations of current knowledge editing methods and highlights the potential of context-based reasoning as a more robust alternative. 1 1 1 https://github.com/ECNU-Text-Computing/Knowledge-Editing-Benchmark.git 1 Introduction Large language models (LLMs) [ 1 , 2 , 3 ] acquire extensive world knowledge [ 4 , 5 , 6 ] and remarkable contextual reasoning abilities [ 7 , 8 ] through large-scale pretraining [ 9 , 10 ] . However, as world knowledge continuously evolves, some of the information encoded in LLMs inevitably becomes outdated or inaccurate [ 11 , 12 ] . Various knowledge editing methods [ 13 , 14 ] have been introduced to enable LLMs to incorporate updated knowledge with minimal parameter modifications or additional cost. The new knowledge pieces to be incorporated are typically collected in textual form. Knowledge editing methods have evolved into five main types, distinguished by how they adjust the parameters of LLMs with reference to collected knowledge texts. Figure 1 illustrates their respective workflows during training and inference. Specifically: (1) Locate-then-edit methods [ 15 , 16 , 17 ] assume that specific knowledge is associated with certain LLM parameters. Thus, they first locate the neurons related to the target knowledge, and then integrates new knowledge by manually editing those parameters. (2) Meta-learning -based methods [ 18 , 19 ] assume that the patterns of parameter changes during knowledge updates can be learned. Accordingly, an editor model is trained to modify the parameters in specific layers of the LLM associated with the target knowledge. (3) Additional parameter -based methods [ 20 , 21 , 22 ] assume that new knowledge can be stored in additional parameters external to the LLM. To achieve this, adapter layers or other auxiliary components are introduced to encode new knowledge while preserving the original parameters of the LLM. (4) In-context learning -based methods [ 23 , 24 ] assume that the target knowledge piece relevant to a question is readily available. By embedding this knowledge directly into the prompts during inference, LLMs can utilize updated information without requiring any parameter modification. (5) External memory -based methods [ 25 , 26 ] store updated knowledge in the form of text, embeddings, hidden states, or even lightweight models. During inference, the most relevant information is retrieved to support LLM reasoning without modifying model parameters. Compared to the highly idealized in-context learning approach, external memory methods offer a more practical solution. Figure 1 : Training and inference workflows for five types of knowledge editing methods. The training process is shown above the red line ; the inference stage is shown below. Despite significant progress in knowledge editing research, a unified evaluation standard has yet to be established, from knowledge format selection, to editing scenarios, then to assess dimensions. Summarized in Table 1 , we systematically review the experimental setups of recent methods. From the perspective of knowledge format , most existing studies rely on fact-based knowledge ( e.g. , triplets), with limited focus on more complex event-level knowledge. Regarding target LLMs , current research primarily focuses on editing instruct LLMs, with little attention given to recent reasoning LLMs. In terms of inference setup for evaluation, most studies use the teacher-forcing strategy ignoring the autoregressive generation setting. Regarding the editing scenarios , many studies do not test in sequential / continuous editing scenario. Furthermore, in evaluating the success of knowledge editing, many studies overlook the portability [ 27 ] of the edited LLMs in real-world applications, particularly when dealing with reverse relations or multi-hop reasoning. These evaluation gaps may obscure critical limitations, impeding a deeper understanding of the field and future advancements. Table 1 : Summary of experimental settings for 12 methods. We specify whether Fact -, Event -based, or Gene ral purpose Datasets are used. Target LLM specifies whether an Instruct LLM or an Reason ing LLM is employed. Infer ence settings show whether Auto regressive setting is applied instead of teacher-forcing. Under Edits , we clarify if Single or Seq uential editing is tested. For evaluation Dimensions , we specify if Rel iability, Gen eralization, Loc ality, and Port ability are considered. Datasets LLM Infer Edits Dimensions Category Method Fact Event Gene. Instruct Reason Auto Single Seq. Rel. Gen. Loc. Port. Locate-then-edit ROME [ 15 ] ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✓ ✓ ✗ MEMIT [ 16 ] ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ PMET [ 17 ] ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ RECT [ 28 ] ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ AlphaEdit [ 29 ] ✓ ✗ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ FT-L [ 15 ] ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✓ ✓ ✗ Meta-learning MEND [ 18 ] ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✓ ✓ ✗ Additional AdaLoRA [ 30 ] ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✓ ✓ ✗ Parameter WISE [ 22 ] ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ In-context IKE [ 23 ] ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ Learning ICE [ 24 ] ✓ ✗ ✗ ✓ ✗ ✓ ✓ ✗ ✓ ✓ ✓ ✓ External Memory GRACE [ 25 ] ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ To this end, we conduct comprehensive benchmarking experiments that simulate practical knowledge editing scenarios. We recognize that real-world applications often require multiple sequential edits as knowledge evolves over time. We adopt a unified autoregressive inference setting, as edited LLMs should not rely on ground truth access as in teacher-forcing settings. Using instruct LLMs as our foundation, we perform knowledge editing with factual data and evaluate performance across four dimensions. Beyond standard single-edit evaluations, we assess the methods’ capability to handle sequential editing scenarios. We systematically compare these effects on both instruct and reasoning-specialized LLMs, while also evaluating the preservation of mathematical reasoning abilities post-editing. To complement our quantitative analysis, we conduct detailed case studies and error analyses. Notably, we introduce Selective Contextual Reasoning (SCR), a straightforward baseline that retrieves relevant knowledge in response to queries and incorporates it as contextual prompts during