ScEdit : Script-based Assessment of Knowledge Editing

Xinye Li, Zunwen Zheng, Qian Zhang, Dekai Zhuang, Jiabao Kang, Liyan Xu, Qingbin Liu, Xi Chen, Zhiying Tu, Dianhui Chu, Dianbo Sui · 2025

Read ScEdit : Script-based Assessment of Knowledge Editing by Xinye Li, Zunwen Zheng, Qian Zhang, Dekai Zhuang, Jiabao Kang, Liyan Xu, Qingbin Liu, Xi Chen, Zhiying Tu, Dianhui Chu, Dianbo Sui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23291.txt.

Introduction

Xinye Li and ten co-authors build a tougher test for editing facts inside language models, in a 2025 paper. Knowledge editing aims to fix a model's facts without retraining. Old tests are so easy that many methods score near perfect, which hides real weakness.

Their benchmark, ScEdit, wraps edits in scripts that models must act out. It covers both invented facts and facts that change over time. Scoring checks single tokens and whole passages. The shift moves testing from what questions to how questions, closer to agent work. Every editing method dropped in score, and passage level checks proved hardest. The benchmark is public for the field.

Catalog record

CREATORXinye Li, Zunwen Zheng, Qian Zhang, Dekai Zhuang, Jiabao Kang, Liyan Xu, Qingbin Liu, Xi Chen, Zhiying Tu, Dianhui Chu, Dianbo Sui
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23291

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ScEdit : Script-based Assessment of Knowledge Editing Xinye Li, Zunwen Zheng, Qian Zhang, Dekai Zhuang, Jiabao Kang, Liyan Xu, Qingbin Liu, Xi Chen, Zhiying Tu, Dianhui Chu, Dianbo Sui Abstract Knowledge Editing (KE) has gained increasing attention, yet current KE tasks remain relatively simple. Under current evaluation frameworks, many editing methods achieve exceptionally high scores, sometimes nearing perfection. However, few studies integrate KE into real-world application scenarios (e.g., recent interest in LLM-as-agent). To support our analysis, we introduce a novel script-based benchmark – ScEdit ( Sc ript-based Knowledge Edit ing Benchmark) – which encompasses both counterfactual and temporal edits. We integrate token-level and text-level evaluation methods, comprehensively analyzing existing KE techniques. The benchmark extends traditional fact-based (“What”-type question) evaluation to action-based (“How”-type question) evaluation. We observe that all KE methods exhibit a drop in performance on established metrics and face challenges on text-level metrics, indicating a challenging task. Our benchmark is available at https://github.com/asdfo123/ScEdit . 1 Introduction Large Language Models (LLMs) have demonstrated outstanding performance in natural language understanding and generation tasks Zhao et al. 2023 . However, these models may produce outdated and erroneous information, leading to non-factual responses Zhang et al. 2023 ; Wang et al. 2024e ; Hernandez et al. 2024 . Given the high cost of retraining LLMs from scratch Sinitsin et al. 2020 , various techniques have been proposed to efficiently update models and support their continual adaptation Zhang et al. 2024 ; Jin et al. 2025 . Among them, Knowledge Editing (KE) has emerged as an increasingly important paradigm for efficiently updating knowledge Meng et al. 2022 ; Yao et al. 2023 ; Wang et al. 2024c . KE methodologies have been developed to incrementally infuse new information or correct existing knowledge without requiring full-scale retraining Mitchell et al. 2022a ; Meng et al. 2022 ; Meng et al. 2023 ; Hartvigsen et al. 2022 ; Zhong et al. 2023 . Figure 1: An example of the script-based assessment of Knowledge Editing (KE). Top: Outdated information generated by the LLM, instructing the user to apply for a visa, thereby misleading them. Bottom: Updated LLM successfully integrates new information, correctly informing the user about the visa-free policy. The conventional evaluation framework for KE largely relies on token-level metrics such as Efficacy , Generalization , and Specificity Meng et al. 2022 . Although these metrics provide a great starting point, they exhibit notable limitations. For instance, Generalization attempts to transcend mere key-value pair memorization by evaluating a model’s capacity to answer rephrased or synonymously expressed questions. However, such evaluations tend to remain in the realm of “What?”-type question transformations, overlooking the broader generalization capabilities of editing methods. Moreover, these three metrics typically gauge KE based on the next few tokens that follow prompts. This approach overlooks the potential for more complex, long-form natural language generation, leaving it largely unaddressed Rosati et al. 2024 . In real-world scenarios, LLMs are increasingly deployed as agents or as core components of multi-agent frameworks that assist users in navigating daily life, making decisions, and performing complex tasks Li et al. 2024 ; Sumers et al. 2024 ; Wang et al. 2024a ; Lal et al. 2024 ; Du et al. 2025a ; Du et al. 2025b . In these roles, users often pose “How”-type questions, which require the models to generate goal-oriented Scripts not only recalling factual information, but applying, generalizing, and reasoning based on that information Lyu et al. 2021 . A Script is a framework describing the sequence of events in a context. Specifically, in the context of KE, the rapidly changing landscape of factual knowledge means that generated Script may become erroneous, potentially misleading users. For example, as illustrated in Figure 1 , a user asks “How does a Chinese citizen travel to Singapore from Beijing?” A pretrained LLM without updated knowledge might suggest applying for a visa, despite Singapore’s new visa exemption policy for Chinese tourists. Such questions necessitate prompt and accurate updates to ensure reliable responses. Existing evaluation approaches, with their focus on token-level factual recall, do not sufficiently address these real-world complexities. To overcome these limitations, this paper introduces a script-based evaluation framework, named ScEdit , assessing KE performance in procedural planning scenarios. ScEdit emphasizes the model’s ability to handle “How?”-type questions and produce coherent, reliable guidance following targeted knowledge updates. A key focus is on how models propagate edited knowledge through script-based procedural planning tasks after editing. We integrate token-level and text-level evaluation, comprehensively analyzing existing KE techniques in both counterfactual and temporal editing tasks. Three LLMs (GPT2-XL Radford et al. 2019 , GPT-J Wang and Komatsuzaki 2021 and Llama 3 AI@Meta 2024 ) are tested in ScEdit . Experimental results on ScEdit reveal that all comparable methods experience an average drop of 27% in the token-level metric S-ES compared to the similar PS metric introduced by Meng et al. 2022 . Moreover, some methods struggle to balance effective editing with maintaining locality in both token-level and text-level evaluations. Even methods that excel in token-level metrics show significant room for improvement in text-level editing performance. These findings highlight the need for further research into KE methods tailored for script-like scenarios. We summarize our contributions of the paper as follows: • Develop a script-based assessment framework that leverages scripts–structured procedural knowledge–to capture a model’s ability to integrate updated facts into complex reasoning and generation tasks. To the best of our knowledge, this is the first attempt to integrate KE into script-based scenarios, presenting a more challenging task compared to existing KE and constrained script generation tasks. • Introduce ScEdit , a novel and challenging script-focused benchmark, accompanied by comprehensive experiments to evaluate models’ ability at both token and text level. Figure 2: Overview of ScEdit . For token-level evaluation, we concatenate the Script Question and Truncated Script to form a cloze-format prompt. For text-level evaluation, we involve automatic and human evaluation. 2 Related Work 2.1 Scripts A script is a structure that describes an appropriate sequence of events in a particular context Schank and Abelson 1975 . Scripts are typically classified into narrative scripts , which describe a sequence of events in a story-like manner Fang et al. 2022 ; Tandon et al. 2020 , and goal-oriented scripts , which outline the steps needed to achieve a specific goal Sancheti and Rudinger 2021 ; Lyu et al. 2021 . Our work aligns with the latter paradigm. Generating high-quality scripts, a longstanding challenge, traditionally involves learning action sequences from narratives by analyzing causal relationships Mooney and DeJong 1985 . Recently, script generation using large language models (LLMs) has become more feasible, with methods such as the over-generate-then-filter approach Yuan et al. 2023 . The script paradigm helps LLMs better understand the temporal order and logical relationships of everyday events. With fine-tuning and post-processing, models demonstrate enhanced generalization abilities in script generation Sancheti and Rudinger 2021 . Smaller models, when trained on high-quality script datasets like CoScript , have shown superior constrained language planning quality compared to LLMs Yuan et al. 2023 . 2.2 Knowledge Editing Knowledge Editing (KE) has emerged as a promising approach to efficiently update LLMs without requiring full retraining Sinitsin et al. 2020 . Many applications and specific tasks also require ongoing

Read the complete text