PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models

Zhaopan Xu, Pengfei Zhou, Weidong Tang, Jiaxin Ai, Wangbo Zhao, Kai Wang, Wenqi Shao · 2025

Read PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models by Zhaopan Xu, Pengfei Zhou, Weidong Tang, Jiaxin Ai, Wangbo Zhao, Kai Wang, Wenqi Shao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12545.txt.

Introduction

Zhaopan Xu and co-authors published a 2025 paper on testing whether AI models can truly forget. Large multimodal models learn from internet data, including private material that may need removal. Machine unlearning promises to delete targeted knowledge without retraining from scratch. Judging whether forgetting worked is tricky: testing on real private data would defeat the purpose.

Their answer is a fully fictitious benchmark, PEBench. Every person and fact in it is invented, so models can be taught the data, asked to forget parts, and then probed safely. Earlier benchmarks lacked this clean setup and covered fewer angles. The dataset lets researchers measure both sides of unlearning: the target knowledge should vanish, while the rest of the model's skill should survive. The authors frame the tool as infrastructure for privacy research, giving a young field a fair and repeatable test.

Catalog record

CREATORZhaopan Xu, Pengfei Zhou, Weidong Tang, Jiaxin Ai, Wangbo Zhao, Kai Wang, Wenqi Shao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-12545

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models Thanks: This work is completed during Zhaopan Xu’s internship at Shanghai Artificial Intelligence Laboratory. This work was supported by the National Key R&D Program of China No.2022ZD0160102 and the National Science Foundation of China under Grant 62476069. Zhaopan Xu and Hongxun Yao are with the School of Computer, Harbin Institute of Technology, China, 150001. (e-mail: [email protected]; [email protected];). Thanks: Pengfei Zhou, Wangbo Zhao and Kai Wang are with the School of Computer, National University of Singapore, Singapore. (e-mail: [email protected]; [email protected]; [email protected];). Thanks: Weidong Tang is with the School of Computer, Xidian University, Singapore, Xian 710000, China (e-mail: [email protected]) Thanks: Jiaxin Ai is with the School of Computer Science, Wuhan Univeristy, Wuhan 430072, China (e-mail: [email protected]) Thanks: Xiaojiang Peng is with the College of Big Data and Internet, Shenzhen Technology University, Shenzhen, 518118, China. (e-mail: pengxiao [email protected]) Thanks: Wenqi Shao and Kaipeng Zhang are with Shanghai Artificial Intelligence Laboratory, Shanghai 200000, China (e-mail: [email protected]; [email protected]). Thanks: †Corresponding author. Zhaopan Xu, Pengfei Zhou, Weidong Tang, Jiaxin Ai, Wangbo Zhao, Kai Wang, Wenqi Shao Abstract Multimodal large language models (MLLMs) have achieved remarkable success in vision-language tasks, but their reliance on vast, internet-sourced data raises significant privacy and security concerns. Machine unlearning (MU) has emerged as a critical technique to address these issues, enabling the selective removal of targeted information from pre-trained models without costly retraining. However, the evaluation of MU for MLLMs remains inadequate. Existing benchmarks often lack a comprehensive scope, focusing narrowly on entities while overlooking the unlearning of broader visual concepts and the inherent semantic coupling between them. To bridge this gap, we introduce, PEBench, a novel benchmark designed to facilitate a thorough assessment of MU in MLLMs. PEBench features a fictitious dataset of personal entities and corresponding event scenes to evaluate unlearning across these distinct yet entangled concepts. We leverage this benchmark to evaluate five MU methods, revealing their unique strengths and weaknesses. Our findings show that unlearning one concept can unintentionally degrade performance on related concepts within the same image, a challenge we term cross-concept interference. Furthermore, we demonstrate the difficulty of unlearning person and event concepts simultaneously and propose an effective method to mitigate these conflicting objectives. The source code and benchmark are publicly available at https://pebench.github.io. I Introduction With the rapid development and widespread application of large language models (LLMs), ethical and safety concerns have drawn much attention, due to the large volumes of data scraped from the internet during the training [ 1 , 2 ] . In response, Machine Unlearning (MU) has been proposed as a remedy [ 3 , 4 , 5 ] . MU is designed to selectively remove the influence and impact of undesirable data from pre-trained models without requiring complete retraining [ 6 ] . Recent progress continues to highlight its increasing effectiveness in addressing these challenges within LLMs [ 7 , 8 ] . Fig. 1: Example of an image of Joe Biden speaking at the White House. Before unlearning (a), MLLMs can generate responses related to various visual concepts ( Person and Event ). The goal of Machine Unlearning (MU) for MLLMs is to selectively forget specific concepts within the model. When the unlearning target is Person (b), the model mistakenly identifies Joe Biden as a different person . When the unlearning target is Event (c), the model misinterprets the speech as a concert . Building on the foundation of LLMs, Multimodal Large Language Models (MLLMs) have achieved remarkable performance in multimodal applications [ 9 , 10 , 11 ] . Consequently, MLLMs inherit the same privacy and security vulnerabilities, including risks of copyright infringement and the retention of sensitive information embedded in their training data [ 12 , 13 ] . Developing effective machine unlearning (MU) is therefore equally critical for MLLMs. Fig. 2: Comparison between previous MU benchmarks and our PEBench for MLLMs. MMUBench (a) utilizes 50 real-world images representing 20 distinct entities, such as popular characters like Mario. CLEAR (b) employs 20 synthetic images for 200 fictitious identities, extending the paradigm of using fictitious data. In contrast, our proposed PEBench (c) features synthetic images encompassing 200 fictitious entities across 40 distinct event scenes, designed to ensure high stylistic consistency across different individuals and various contexts. While numerous benchmarks have been proposed to advance MU in LLMs [ 14 , 15 ] , the landscape for MLLMs remains underdeveloped. Existing MLLM-specific benchmarks, such as MMUBench [ 16 ] (Fig. 2 (a)) and CLEAR [ 17 ] (Fig. 2 (b)), are constrained to unlearning entities. This narrow focus overlooks the broader spectrum of visual concepts, which includes not only concrete entities like individuals but also general contexts such as event scenes (Fig. 1 (a)). Critically, current benchmarks fail to account for the coupling of these concepts, as a single image inherently contains semantically entangled elements. This raises a question: when forgetting an entity, does it affect other concepts within the same image (such as event scenes)? Such limitations lead to an insufficient evaluation of an MU method’s true efficacy, which motivates our work to develop a more comprehensive benchmark. We categorize multimodal unlearning targets into two types based on the scope of visual concepts: Person (Fig. 1 (b)) and Event (Fig. 1 (c)) . This distinction is motivated by real-world challenges that raise significant privacy and safety concerns in multimodal learning. Specifically, the Person category relates to protecting personal privacy and intellectual property, while the Event category involves removing potentially harmful or illegal content, such as scenes depicting fake news. Accordingly, we introduce a new benchmark named PEBench , which is explicitly designed to assess the unlearning of both P erson and E vent concepts. PEBench is constructed using synthetic data, offering two primary advantages: (1) Establish a reliable upper bound for unlearning performance. While retraining a model from scratch after removing target data represents the gold standard for exact unlearning [ 18 , 19 ] , this approach is computationally infeasible for LLMs. TOFU [ 15 ] addresses this issue by introducing fictitious data that was never seen during pretraining, allowing fine-tuned LLMs to approximate the behavior of retraining. Specifically, TOFU features 200 fictitious author profiles, each described by attributes such as name, birthplace, parents’ names, and occupations, including 4,000 question-answer pairs for evaluating LLMs’ unlearning. CLEAR [ 17 ] extends TOFU to the MLLMs by generating consistent synthetic images associated with the TOFU authors over 200 fictitious identities. Following this paradigm, PEBench also comprises 200 fictitious individuals, but significantly expands its complexity by linking each individual to 40 distinct event scenes, creating a dataset of 8,000 images for more nuanced evaluations. (2) Ensure controllable data quality for satisfying both generality and scope evaluation. Generality assesses whether a model can forget a concept across diverse samples [ 20 ] , whereas scope measures if only the intended target is forgotten while preserving unrelated information [ 21 , 22 ] . PEBench is therefore synthetically generated to ensure both high intra-target consistency (for evaluating generality) and deliberate inter-concept coupling (for evaluating scope). This design facilitates a reliable and fine-grained assessment of unlearning efficacy and its potential side effects. We benchmark 5 MU methods, providing new

Read the complete text