K-Edit: Language Model Editing with Contextual Knowledge Awareness

Elan Markowitz, Anil Ramakrishna, Ninareh Mehrabi, Charith Peris, Rahul Gupta, Kai-Wei Chang, Aram Galstyan · 2025

Read K-Edit: Language Model Editing with Contextual Knowledge Awareness by Elan Markowitz, Anil Ramakrishna, Ninareh Mehrabi, Charith Peris, Rahul Gupta, Kai-Wei Chang, Aram Galstyan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10626.txt.

Introduction

Elan Markowitz and six co-authors present K-Edit, a way to patch a language model's facts while keeping the facts' neighbors consistent. Their paper appeared in two thousand twenty-five. Editing tools can swap one stored fact for another without retraining. But a fact rarely stands alone: change a capital and the answers about its country should stay coherent.

K-Edit pulls the needed context from a knowledge graph. When an edge in the graph changes, the method derives the related edits that keep the web consistent. Those extra edits are applied to the model alongside the main one. Tests show clear gains on questions that hop across several facts. The base editing skill and its scale did not suffer. The work suits models that must track a changing world. It treats knowledge as a web, not a list.

Catalog record

CREATORElan Markowitz, Anil Ramakrishna, Ninareh Mehrabi, Charith Peris, Rahul Gupta, Kai-Wei Chang, Aram Galstyan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10626

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

K-Edit: Language Model Editing with Contextual Knowledge Awareness Elan Markowitz, Anil Ramakrishna, Ninareh Mehrabi, Charith Peris, Rahul Gupta, Kai-Wei Chang, Aram Galstyan Abstract As the world changes, we need to be able to update our models and correct false information without costly retraining. Knowledge-based model editing enables precise modifications to the weights of large language models in order to modify the information encoded within. Recent approaches have seen success in enabling recall of edited information for thousands of edits at once. However, these approaches fail to produce edits that account for associated contextual information. We present K-Edit, an effective approach to generating contextually consistent knowledge edits. By using knowledge graphs, which maintain contextual consistency when an edge is edited, we are able to generate additional contextual edits that ensure consistency of related information in the language model. Our experiments demonstrate significant improvements in multi-hop question answering while maintaining the general effectiveness and scalability of model edits. 1 Introduction Figure 1: Example of how K-Edit improves model editing. Whereas existing methods edit the model so that it recalls the correct updated information, that new information is not associated with contextual information. The result is that the LLM is left with an inconsistent world model in which “The Prime Minister is Rishi Sunak”, and “Rishi Sunak is married to Akshata Murty”, but “The Prime Minister’s spouse is Carrie Johnson”. K-Edit fixes this by querying a knowledge graph (such as Wikidata) for contextual information, and then turning that information into contextual edits , and applying it to the LLM to improve multi-hop reasoning and consistency. The ability to pinpoint and edit knowledge within the parameters of a Large Language Model (LLM) is a powerful tool for maintaining alignment with an ever-changing world. Existing approaches to direct model editing make shallow edits that fail to integrate with contextual knowledge the model possesses ( Zhong et al., 2023 ) . This work proposes and explores a simple solution to better enhance knowledge reasoning over edits by including contextual and associated knowledge in the editing process. We use the contextual consistency of knowledge graphs when edited to improve the consistency of language models under direct model editing. Direct Model Editing (or Knowledge-based Model Editing) aims to change the implicit knowledge in an LLM’s weights with no other changes to how the model is used ( Wang et al., 2023b ; Mazzia et al., 2023 ; Yao et al., 2023 ) . Prior methods have focused on the ability to edit and recall a new fact without affecting neighboring facts or general generation ability ( Zhu et al., 2020 ; Mitchell et al., 2021 ; Meng et al., 2022a ; Meng et al., 2022b ) . Recently, however, researchers have begun to question the effectiveness of these edits through benchmarks evaluating the ability of models to use these edits for multi-hop reasoning ( Zhong et al., 2023 ) . For example, if we edit the British Prime Minister to be Rishi Sunak instead of Boris Johnson , we would expect the model to be able to answer Who is the British Prime Minister’s wife? with the correct answer Akshata Murty . Existing direct model editing techniques largely fail in this regard. We address this issue by utilizing the relational information in knowledge graphs. In this work, we propose K-Edit, a simple, yet effective, approach of adding more contextual knowledge into the editing process. In contrast to text (and thus LLMs), knowledge graphs are very simple to edit. Knowledge graphs are graph databases encoding the relations between real-world entities. Since the structure of knowledge is explicit in a knowledge graph, editing a single edge immediately updates connected associations as well. We propose a method that incorporates these automatic associations into the editing process. After an edit is made, we extract related contextual information from a knowledge graph. We then construct contextual edits that ensure the context is ingrained in addition to the initial edit. Figure 1 shows the advantage of this approach. Our results show that this method significantly helps with the primary goal, as measured by significant improvement on multi-hop reasoning over edited information. We even show that such edits improve multi-hop reasoning even when the required reasoning path does not directly appear in the context. Our results also indicate that there is no cost to quality as measured by other knowledge editing metrics, and that thousands of such edits can be applied without degradation compared to baselines. Our contributions are 1. K-Edit, a new direct model editing algorithm for improving contextual and associated knowledge awareness through incorporation of knowledge graph data. 2. Experimental evaluation that shows improved multi-hop reasoning with K-Edit and its ability to generalize. 3. Additional experiments and ablations showing no degradation to other knowledge editing metrics and the ability to apply thousands of such edits at a time. 2 Background 2.1 Knowledge Graphs Knowledge graphs encode real world entities and the relations between them. The most prominent knowledge graphs contain billions of edges covering a vast array of subjects, such as Wikidata ( Vrandečić and Krötzsch, 2014 ) and DBPedia ( Lehmann et al., 2015 ) . We define a knowledge graph \mathcal{K}=(\mathcal{E},\mathcal{R},\mathcal{T}) where \mathcal{E} is the set of entities, \mathcal{R} is the set of relations, and \mathcal{T} is the set of edges (s,r,o)\in\mathcal{E}\times\mathcal{R}\times\mathcal{E} that represent true relationship in the knowledge base between subject s and o . For example, if (\textit{Blank Space, performer, Taylor Swift})\in\mathcal{T} , then the statement “The performer of Blank Space is Taylor Swift” is considered true. One feature of knowledge graphs is that if an edge is updated, all multi-hop relations associated with that edge will be correct without any additional explicit updates. 2.2 Knowledge-based Model Editing We define some background notation and concepts for knowledge-based model editing. Here we deal with edits of the form, (s,r,o\rightarrow o^{*}) , meaning we want to edit subject s such that the relationship r now refers to object o* instead of o . We define a text prompt t_{r}(s) for the edit (also referred to as a cloze statement) in which o^{*} is the desired output, e.g. t_{\text{prime minister}}(\text{{United Kingdom}}) = “The Prime Minister of { the United Kingdom } is”. In this example the United Kingdom is the entity for which we want to modify the Prime Minister association. In general, knowledge-based model editing approaches seek to modify the model parameters \theta such that the prompt t_{r}(s) will lead to the generation of text o^{*} . 2.3 Batch Editing In order for our approach, K-Edit, to succeed, we rely on the ability to make edits in a batch. Our approach works with any such algorithm. For our experiments, we use MEMIT, a state-of-the-art batch editing algorithm that can scale to thousands of edits at once ( Meng et al., 2022b ) . MEMIT is the best option to use with K-Edit, and the only option that can scale to large numbers of edits. MEMIT has three key properties: (1) it computes the desired model parameter changes for thousands of edits independently, (2) it precisely targets associations of the subjects to be edited, and (3) it can apply all edits together without significant model degradation. Further background details on how MEMIT

Read the complete text