Extracting Unlearned Information from LLMs with Activation Steering

Atakan Seyitoğlu, Aleksei Kuvshinov, Leo Schwinn, Stephan Günnemann · 2024

Read Extracting Unlearned Information from LLMs with Activation Steering by Atakan Seyitoğlu, Aleksei Kuvshinov, Leo Schwinn, Stephan Günnemann (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02631.txt.

Description

Abstract An unintended consequence of the vast pretraining of Large Language Models (LLMs) is the verbatim memorization of fragments of their training data, which may contain sensitive or copyrighted information. In recent years, unlearning has emerged as a solution to effectively remove sensitive knowledge from models after training.

Catalog record

CREATORAtakan Seyitoğlu, Aleksei Kuvshinov, Leo Schwinn, Stephan Günnemann
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-02631

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Extracting Unlearned Information from LLMs with Activation Steering Atakan Seyitoğlu, Aleksei Kuvshinov, Leo Schwinn, Stephan Günnemann Abstract An unintended consequence of the vast pretraining of Large Language Models (LLMs) is the verbatim memorization of fragments of their training data, which may contain sensitive or copyrighted information. In recent years, unlearning has emerged as a solution to effectively remove sensitive knowledge from models after training. Yet, recent work has shown that supposedly deleted information can still be extracted by malicious actors through various attacks. Still, current attacks retrieve sets of possible candidate generations and are unable to pinpoint the output that contains the actual target information. We propose activation steering as a method for exact information retrieval from unlearned LLMs. We introduce a novel approach to generating steering vectors, named Anonymized Activation Steering. Additionally, we develop a simple word frequency method to pinpoint the correct answer among a set of candidates when retrieving unlearned information. Our evaluation across multiple unlearning techniques and datasets demonstrates that activation steering successfully recovers general knowledge (e.g., widely known fictional characters) while revealing limitations in retrieving specific information (e.g., details about non-public individuals). Overall, our results demonstrate that exact information retrieval from unlearned models is possible, highlighting a severe vulnerability of current unlearning techniques. 1 Introduction Large language Models (LLMs) are trained on vast amounts of text data curated from diverse sources. The extensive training process enables LLMs to generate high-quality responses to a wide range of topics. However, an unintended consequence of this approach is the verbatim memorization of fragments of their training data, which may contain sensitive information. Here, the scale of the training data prevents the reliable identification and removal of all instances of private or copyrighted information, such as personal addresses, passages from copyrighted books, or proprietary code snippets. Regulations, such as the EU’s General Data Protection Regulation (GDPR) ( Voigt & Von dem Bussche 2017 ) , aim to give individuals control over their personal information through measures, such as the “Right to erasure” , requiring companies to delete user data upon request ( Zhang et al. 2023a ) . However, as retraining large models from scratch is impractical to delete user information, unlearning has emerged as an alternative solution to effectively remove knowledge from models while preserving their overall performance ( Maini et al. 2024 ) . Evaluating the success of an unlearning method is a difficult task. Even if the model does not directly respond correctly to questions about the unlearned topic, it doesn’t necessarily mean the concept is fully forgotten. Several existing approaches aim to recover unlearned or deleted information ( Lynch et al. 2024 ) . Existing works consider an attack successful if the correct answer is present among the candidates but are unable to pinpoint the answer containing the correct information ( Patil et al. 2024 ; Schwinn et al. 2024 ) . In contrast, we argue that for real-world applications, information retrieval systems should incorporate scoring mechanisms that account for the accuracy of selecting the correct answer, which results in a more accurate assessment of leakage risk during deployment. Our contributions in this paper are as follows: • We introduce a novel attack based on activation steering (see § 2 ) against LLM unlearning. To this end, we deploy a novel way of generating pairs of prompts to calculate activation differences – Anonymized Activation Steering. • We evaluate our approach on three unlearning methods and three different datasets. Contrary to existing attacks, we demonstrate that our proposed approach can retrieve unlearned information in an exact manner pinpointing the correct answer among a set of candidates with high accuracy. • We investigate failure scenarios for our approach and find that while exact information retrieval is successful on general knowledge (i.e., models unlearned on Harry Potter) it fails for specific, less known information. • We provide a new dataset based on existing work ( Schwinn et al. 2024 ) for Harry Potter information retrieval, enabling more accurate assessments of unlearning methods. In this work, we demonstrate the power of activation steering as the evaluation tool for targeted unlearning of LLMs. Our results highlight its effectiveness for broad topics, such as removing copyright-related information, while revealing its limitations when applied to more specific knowledge. By exploring these strengths and constraints, we provide deeper insights into how activation steering affects the behavior of unlearned LLMs, contributing to the development of safe LLMs. 2 Related Work LLM unlearning is a popular research topic due to its efficiency compared to retraining from scratch. Eldan & Russinovich 2023 make a model forget the entire Harry Potter franchise, while Li et al. 2024 unlearn harmful information in biology, chemistry, and cyber-security domains. Apart from unlearning entire domains of information, methods exist that solve the privacy issue of LLMs, deleting personal information at request ( Jang et al. 2023 ; Wu et al. 2023 ) . Meng et al. 2022 replace more specific information from an LLM, unlearning a single sentence at a time. To effectively test unlearning methods, a clear benchmark is essential. In this context, Maini et al. 2024 introduce the TOFU dataset, which consists of synthetic data about fictitious authors. Because the data is artificial, it allows for direct comparison between a model trained on this dataset that has undergone unlearning, against a baseline model, that is guaranteed to never have seen this dataset during training. Similarly, Li et al. 2024 provide the WMDP dataset, designed specifically for benchmarking unlearning techniques. This dataset contains harmful information, enabling evaluation of a model’s ability to effectively forget potentially dangerous content. Similar to retrieving information from an unlearned model, researchers perform attacks on standard models that refuse to answer harmful prompts. A lot of research in this area is in a “jailbreak” setting where the model already knows the information but refuses to answer ( Chu et al. 2024 ) . Shi et al. 2024 identify if a given data was seen during training to recover private information. Attacking an unlearned model is similar, as the goal is to recover information, but in this case, the model does not refuse to answer but outputs seemingly random answers or replaced information such as in unlearning done with ROME method by Meng et al. 2022 . Patil et al. 2024 show that ROME and other unlearning methods such as MEMIT ( Meng et al. 2023 ) are prone to attacks, and the information is not truly deleted. Activation steering is a technique for manipulating LLMs’ latent space and guiding the generated responses in the desired direction. This method is applied to overcome refused prompts ( Arditi et al. 2024 ; Rimsky et al. 2024 ) , reduce toxicity ( Turner et al. 2023 ) , enhance truthfulness ( Li et al. 2023a ) , and adjust the tone of responses ( von Rütte et al. 2024 ) in LLMs. It works by calculating a “steering vector” that reflects the target direction, derived from pairs of inputs that differ based on the intended direction of behavior (e.g., toxic and non-toxic pairs). The steering vector is obtained through simple subtraction or more advanced techniques like Principal Component Analysis (PCA) or Logistic Regression ( Tigges et al. 2023 )

Read the complete text