Classifier-free guidance in LLMs Safety

Roman Smirnov · 2024

Read Classifier-free guidance in LLMs Safety by Roman Smirnov (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06846.txt.

Introduction

Roman Smirnov, writing alone in 2024, describes a way to make a large language model forget trained content without keeping a set of good examples to protect. Forgetting, called unlearning, is needed when models hold private or unwanted text. His method trains the model on synthetic replacement data using a reinforcement learning approach named ORPO.

The twist is to train with a guidance method in mind and then apply that guidance when the model runs. Classifier-free guidance steers output by contrasting two model states, and Smirnov adapts it for the forgetting task. He reports much stronger unlearning with no clear loss in the model's general skill. The work is an extended version of a prize-winning entry to a NeurIPS 2024 competition on language model privacy.

Catalog record

CREATORRoman Smirnov
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06846

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Classifier-free guidance in LLMs Safety Roman Smirnov Abstract This article is an extended version of the NeurIPS 2024 LLM-PC submission that was awarded the second prize. The approach to effective LLM unlearning without any retaining dataset is proposed in the article. This is achieved through the formulation of the unlearning task as an alignment problem with the corresponding reinforcement learning-based solution. Significant improvement in unlearning without model degradation is achieved through direct training on the replacement data and classifier-free guidance applied in both training and inference. Sections 4 and 5 of the article were added after the NeurIPS 2024 LLM-PC competition and are focused on data ablation study and enhancements to classifier-free guidance implementation for large language models. 1 Introduction LLM unlearning is an increasingly important research area, gaining attention as large language models are applied across various industries and social contexts. Unlearning can be applied to unlearn specific behaviors, e.g. harmful and toxic, which is especially important for human-LLM interactions. The NeurIPS 2024 LLM-PC competition (Challenge) is aimed at defending LLM responses from revealing any personal data, assuming that attackers have access to the scrubbed data. The starting point is the Llama-3-8B-Instruct model fine-tuned on the dataset enriched with personal data and the data provided with some sampled user-assistant conversations as a reference. The competition page and initial data: LLM-PC github . The code for the approach described in this article is in the project github repository . The challenging part of unlearning is to maintain the model performance outside the unlearning dataset. Recent approaches like [ 1 ] and [ 2 ] use a retaining dataset for that. Some methods, e.g. [ 2 ] and [ 3 ] , use the loss on the retaining dataset as a part of the training loss that may introduce additional bias towards it. The approach introduced in this article is inspired by the idea that unlearning of such concepts like harmful behavior or personal data usage in the answers lies in a field of LLM alignment where maintaining base model performance is achieved through KL-divergence or its approximations without any retaining dataset through reinforcement learning. To implement reinforcement learning approach, the reward model is required (in the context when we have the samples to forget, the reward model is still required to classify chosen and rejected samples for DPO-style tuning). A simple named entity recognition model can be used in the case of personal data unlearning as the reward model. However, "good" answers are still required to do DPO-style tuning. Following the [ 2 ] these "good" answers can be generated using external API-based LLMs (with some modifications of the generation task that are covered in Section 2.1). Using such answers in training makes the training objective more direct and reliable compared to the scenario when the gradient ascent is applied with negative examples only. The article presents a method for LLM unlearning that does not require a retraining dataset. 2 Methodology The Challenge is about making the model robust to the personal data leakage, when the attackers have access to the scrubbed data. This means that the model should not generate responses containing personal data (PII), even when personal data or specific patterns are present in the dialogue history or the prompt. The proposed approach consists of four components: • Models subtraction with the task vectors; • Extensive data generation and filtration; • Supervised fine-tuning and DPO-style tuning; • Inference modification to improve model’s robustness. 2.1 Model preparation Following the forgetting via negation method proposed in [ 4 ] and [ 5 ] we can build the following pipeline: we have the base model and we generate data with extensive PII usage in responses, after that we do supervised fine-tuning of the base model with this data, so we get the modified (reinforced) model. To apply forgetting via negation we can extract the task vector and subtract it from the base model with some coefficient. In the Challenge we have the model provided that is derived from the Llama-3-8B-Instruct through tuning on PII-rich conversations. To apply forgetting via negation the delta weights should be calculated and subtracted from the base model (Llama-3-8B-Instruct) with some coefficient - 0.5 was used in the experiments. The coefficient 0.5 was used because the model received with higher coefficients, e.g. 1.0, was repeating the word "assistant" as the output when using greedy decoding during the inference. The model we got after subtraction is called Model-sub further, while the provided in the Challenge model - Model-ch . Following the approach in [ 5 ] , the ReLU activation can be applied to the delta vector. Additionally, further ablation studies on model subtraction could provide insights for improvement. 2.2 Data generation The material provided in the Challenge included data samples rich in PII. The schema of the provided data samples can be represented by the following dialogue template: System: default User: Abc [PII] abd Assistant: Abc [PII] abc abc [PII] abc User: … From each dialog I got the same number of samples as there are PIIs in the assistant’s responses - the model should be trained only on the assistant answers and the previous context before the answer can include PII, moreover a target text in a sample can be just a part of the assistant’s answer: the input can be "…Assistant: Abc [PII] abc abc" and the output - "[NOT PII] abc…". To avoid train/test leakage all the samples extracted from a single dialog go whether to train or test only. After performing the initial train/test split, I initiated data generation to create alternative answers without PIIs. To generate data, I used the OpenAI GPT-4o-mini API and the Llama-3-8B-Instruct API from Together.ai. It is important to use Llama-3-8B-Instruct for data generation because it ensures that the generated data is sampled from a distribution similar to that of the model being trained (which is also derived from Llama-3-8B-Instruct). This alignment facilitates faster convergence during training. The following data generation approaches were used: • Llama-3-8B-Instruct with additional system prompt "Avoid using any personal data in the answers!" and 0.7 temperature in completion mode (to cover the case when we want to predict a part of the Assistant’s answer having the beginning of the answer as an input); • Llama-3-8B-Instruct with additional system prompt "Avoid using any personal data in the answers!" and 0.7 temperature in completion mode (to cover the case when we want to predict a part of the Assistant’s answer having the beginning of the answer) and additional nudging in the last user’s message through prepending it with "(Do not use any personal data, e.g. names, locations or any other personal data in your answer even if it was used in the dialog)"; • GPT-4o-mini with additional system prompt "Avoid using any personal data in the answers!" and 0.7 temperature and additional nudging in the last user’s message through prepending it with "(Do not use any personal data, e.g. names, locations or any other personal data in your answer even if it was used in the dialog)"; • GPT-4o-mini with additional system prompt "Avoid using any personal data in the answers!" and 0.7 temperature, additional nudging in the last user’s message through prepending it with

Read the complete text