BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing

Dongliang Guo, Mengxuan Hu, Zihan Guan, Thomas Hartvigsen, Sheng Li · 2025

Read BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing by Dongliang Guo, Mengxuan Hu, Zihan Guan, Thomas Hartvigsen, Sheng Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01343.txt.

Introduction

Editing a fact inside a multimodal AI model is tricky: change too little and the fact stays wrong, change too much and unrelated knowledge breaks. Dongliang Guo and co-authors name that tension the generality-locality trade-off. Different facts deserve different blast radii, yet older editors treat them all alike.

Their method, BalancEdit, first probes each fact with positive and negative examples to map its true influence scope. The edit then lands in a small, discrete codebook inside the model's latent space, leaving the base weights untouched. A new dataset, OKEDIT, grades editors on exactly this balance. Tests in 2025 show strong edits with minimal collateral damage.

Catalog record

CREATORDongliang Guo, Mengxuan Hu, Zihan Guan, Thomas Hartvigsen, Sheng Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01343

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing Dongliang Guo, Mengxuan Hu, Zihan Guan, Thomas Hartvigsen, Sheng Li Abstract Large multi-modal models inevitably decay over time as facts update and previously learned information becomes outdated. Traditional approaches such as fine-tuning are often impractical for updating these models due to their size and complexity. Instead, direct knowledge editing within the models presents a more viable solution. Current model editing techniques, however, typically overlook the unique influence ranges of different facts, leading to compromised model performance in terms of both generality and locality. To address this issue, we introduce the concept of the generality-locality trade-off in multi-modal model editing. We develop a new model editing dataset named OKEDIT, specifically designed to effectively evaluate this trade-off. Building on this foundation, we propose BalancEdit , a novel method for balanced model editing that dynamically achieves an optimal balance between generality and locality. BalancEdit utilizes a unique mechanism that generates both positive and negative samples for each fact to accurately determine its influence scope and incorporates these insights into the model’s latent space using a discrete, localized codebook of edits, without modifying the underlying model weights. To our knowledge, this is the first approach explicitly addressing the generality-locality trade-off in multi-modal model editing. Our comprehensive results confirm the effectiveness of BalancEdit, demonstrating minimal trade-offs while maintaining robust editing capabilities. Our code and dataset are available at https://github.com/donglgcn/BalancEdit/tree/MMOKVQA . 1 Introduction Large multi-modal models ( Zhu et al., 2023 ; Radford et al., 2021 ; Li et al., 2023 ; Liu et al., 2023a ; Rombach et al., 2022 ) have recently brought about significant advancements in artificial intelligence, demonstrating impressive results in tasks such as Visual Question Answering (VQA) ( Antol et al., 2015 ) . However, these models are susceptible to issues like hallucination ( Rawte et al., 2023 ) and fact alteration ( De Cao et al., 2021 ; Guo et al., 2024 ; Zhu et al., 2024 ) . After deployment, these models may generate numerous errors, leading to potential problems like the propagation of hate speech ( Guan et al., 2025b ; Guan et al., 2025a ; Hu et al., 2025 ; Hu et al., 2024 ) or the dissemination of outdated factual information. Given these challenges, it is critical to continually update and maintain these large multi-modal models to ensure their accuracy and relevance. While retraining or fine-tuning can update a model’s knowledge, it is often infeasible to frequently edit individual facts due to the high computational costs involved. Fortunately, model editing techniques ( Hartvigsen et al., 2024 ; Mitchell et al., 2021 ; Zheng et al., 2023 ) provide a promising approach to implementing cost-effective, targeted updates to large pretrained models. These techniques typically involve injecting new layers or modifying weights to alter the knowledge embedded in language models. A successful edit generally exhibits three characteristics ( Mitchell et al., 2021 ; Huang et al., 2023 ) : reliability , which ensures the output changes to the target answer for the same question; locality , which leaves unrelated knowledge and outputs unchanged; and generality , which produces the correct answer for all questions within the influence scope. As illustrated in Fig. 1 , each fact has its own influence scope. For instance, if we wish to edit the name of a specific cat, the influence scope would be confined to that particular cat. If we aim to edit the name of a cat breed, the influence scope would extend to all cats within that breed. However, if we intend to edit the name of a species, the influence scope would encompass all cats. Consequently, we should consider each fact individually and dynamically to determine the appropriate influence scope. Figure 1: Illustration of various influence scope However, current model editing techniques often overlook the dynamic nature of the influence scope. Some methods treat all influence scopes as if they are large and uniform, while others focus solely on a specific edit. For instance, IKE ( Zheng et al., 2023 ) employs in-context learning to edit knowledge, using the closest piece of knowledge as a prompt to guide the language model. This approach causes the language model to rephrase the nearest fact, resulting in an oversized influence scope. Conversely, GRACE ( Hartvigsen et al., 2024 ) , a lifelong model editing method, assumes that each edit has a small and similar influence range, leading to limited generality. Consider an example where we aim to edit a “fact” that HP computers have been renamed Lenovo, as shown in Table 1 . Ideally, model editing should update the answer from HP to Lenovo whenever it encounters an image of a HP computer, while leaving the answer unchanged for other brands. However, existing model editing techniques, such as IKE ( Zheng et al., 2023 ) and MEND ( Mitchell et al., 2021 ) , may achieve the target edit but neglect the influence scope, inadvertently editing other brands as well. Even when presented with a black image, these models may still output the new answer, leading to hallucination. On the other hand, while GRACE ( Hartvigsen et al., 2024 ) maintains the backbone model’s answer for unrelated images, it fails to edit the knowledge to the desired scope. These observations suggest that existing multi-modal model editing methods struggle to dynamically adjust the influence scope of a knowledge edit, and to balance generality and locality effectively. To address this issue, we first create a dataset designed to evaluate the trade-off between generality and locality in model editing techniques. We then introduce an efficient multi-modal model editing method named BalancEdit , which dynamically balances this trade-off with minimal computational costs. Specifically, we incorporate an adapter into a chosen layer of a vision language model without altering its weights. This adaptor modifies layer-to-layer transformations for select inputs. By caching embeddings for input edits and the updated knowledge transformation layer, BalancEdit functions as a codebook where edits are stored. To strike a balance between generality and locality, we generate the corresponding positive and negative samples for each edit. The model’s semantic similarity in its latent space can be visualized as dynamic spheres around cached edits, with the radius determined by the distance between positive and negative samples. By adjusting the radius over time, BalancEdit allows for immediate edits, retains previous edits, and preserves correct model behaviors. Furthermore, since BalancEdit’s codebooks do not alter model weights and are fully model-agnostic, they also pave the way for plug-and-play, cost-effective model editing. This is particularly useful for making critical spot-fixes between larger retraining efforts. Our contributions are as follows: 1) We first formulate the generality-locality trade-off in multi-modal model editing and build a dataset named OKEDIT to empirically demonstrate it. 2) We introduce BalancEdit, an efficient method for multi-modal model editing that dynamically and effectively balances generality and locality without requiring training data beyond individual edits. 3) Our experiments reveal that BalancEdit outperforms baseline models and consistently achieves SOTA performance across a range of metrics. Original Image Related Image Unrelated Image Black Image Question What brand is this computer? Target hp \rightarrow lenovo Base hp hp dell black IKE lenovo lenovo lenovo lenovo MEND lenovo lenovo lenovo lenovo GRACE

Read the complete text