Rule Based Rewards for Language Model Safety
Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, Lilian Weng · 2024
Read Rule Based Rewards for Language Model Safety by Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, Lilian Weng (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01111.txt.
Description
Abstract Reinforcement learning based fine-tuning of large language models (LLMs) on human preferences has been shown to enhance both their capabilities and safety behavior. However, in cases related to safety, without precise instructions to human annotators, the data collected may cause the model to become overly cautious, or to respond in an undesirable style, such as being judgmental.
Catalog record
| CREATOR | Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, Lilian Weng |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01111
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rule Based Rewards for Language Model Safety Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, Lilian Weng Abstract Reinforcement learning based fine-tuning of large language models (LLMs) on human preferences has been shown to enhance both their capabilities and safety behavior. However, in cases related to safety, without precise instructions to human annotators, the data collected may cause the model to become overly cautious, or to respond in an undesirable style, such as being judgmental. Additionally, as model capabilities and usage patterns evolve, there may be a costly need to add or relabel data to modify safety behavior. We propose a novel preference modeling approach that utilizes AI feedback and only requires a small amount of human data. Our method, Rule Based Rewards (RBR), uses a collection of rules for desired or undesired behaviors (e.g. refusals should not be judgmental ) along with a LLM grader. In contrast to prior methods using AI feedback, our method uses fine-grained, composable, LLM-graded few-shot prompts as reward directly in RL training, resulting in greater control, accuracy and ease of updating. We show that RBRs are an effective training method, achieving an F1 score of 97.1, compared to a human-feedback baseline of 91.7, resulting in much higher safety-behavior accuracy through better balancing usefulness and safety. OpenAI Content may include language related to racism, erotic themes, self-harm, or other offensive material. 1 Introduction As large language models (LLMs) grow in capabilities and prevalence, it becomes increasingly important to ensure their safety and alignment. Much recent work has focused on using human preference data to align models, such as the line of work on reinforcement learning from human feedback (RLHF) [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] . However, there are many challenges in using human feedback alone to achieve a target safety specification. Collecting and maintaining human data for model safety is often costly and time-consuming, and the data can become outdated as safety guidelines evolve with model capability improvements or changes in user behaviors. Even when requirements are relatively stable, they can still be hard to convey to annotators. This is especially the case for safety, where desired model responses are complex, requiring nuance on whether and how to respond to requests. If instructions are underspecified, annotators may have to rely on personal biases, leading to unintended model behaviors, such as becoming overly cautious, or it responding in an undesirable style (e.g. being judgmental). For example, some annotators in one of our experiments, when ranking possible responses to user requests pertaining to self-harm, favored completions that referred the user to a US suicide hotline phone number, which would not have helped users in other regions. Fixing such issues often requires relabeling or collecting new data, which is expensive and time consuming. To address these issues, methods that use AI feedback [ 9 , 10 , 11 , 12 ] have recently gained popularity, most prominently Constitutional AI [ 10 ] . These methods use AI feedback to synthetically generate training data to combine with the human data for the supervised fine-tuning (SFT) and reward model (RM) training steps. However, in Bai et al. [ 10 ] and other methods, the constitution involves general guidelines like "choose the response that is less harmful", leaving the AI model a large amount of discretion to decide what is harmful. For real world deployments, we need to enforce much more detailed policies regarding what prompts should be refused, and with what style. In this work, we introduce a novel AI feedback method that allows for detailed human specification of desired model responses, similar to instructions one would give to a human annotator. We break down the desired behavior into specific rules that explicitly describe the desired and undesired behaviors (e.g. "refusals should contain a short apology" , "refusals should not be judgemental toward the user" , , "responses to self-harm conversations should contain an empathetic apology that acknowledges the user’s emotional state." ). This separation into rules is similar to the human feedback method proposed in Sparrow [ 5 ] , however we focus on utilizing AI feedback as opposed to human feedback. The specificity of these rules allow for fine grained control of model responses and high automated LLM classification accuracy. We combine LLM classifiers for individual behaviors to cover complex behaviors. Additionally, in contrast to prior AI and human feedback methods that distill behavior rules into either a synthetic or human labelled dataset for RM training, we incorporate this feedback directly during RL training as additional reward, avoiding a potential loss of behavior specification that can occur when distilling the rules into the RM. Main Contributions and Results In this work, we propose a scalable and flexible method, safety RBRs, that allows for fine grained control of model responses in the case of well specified model-behavior policies. 1. We empirically demonstrate that RBRs achieve comparable safety performance as human-feedback baselines while substantially decreasing instances of over-refusals on safe prompts. Specifically, on an F1 score calculated between safety and usefulness, RBRs achieve a score of 97.1, compared to a human-feedback baseline of 91.7 and a helpful-baseline of 95.8. 2. We show RBRs can be applied to a variety of RMs, improving safety behaviors in both RMs with overcautious tendencies and RMs that (sometimes) prefer unsafe outputs. 3. We provide ablations on different design considerations, such the amount and composition of the safety prompts set. 2 Related Works Reinforcement Learning from Human Feedback (RLHF): Research in RLHF methods [ 1 , 2 , 3 , 7 ] demonstrates the efficacy of human annotations in steering model behavior. A subset [ 4 , 8 , 13 ] of this RLHF research considers achieving better safety behavior through methods such as separating out signals of helpfulness and harmlessness. Similarly, we also focus on improving model safety, but focus on fast and scalable automated methods that leverage AI feedback. Most related to our work, Sparrow [ 5 ] proposes a novel approach to RLHF which trains a second rule-conditioned RM to detect potential rule violations. Like Sparrow, we also use rules, but we have a few key differences. Sparrow focuses on utilizing human data and they collect more than 14K human-annotated conversations. We instead focus on utilizing AI feedback. Additionally, our approach involves fitting a model to ensure that the final reward effectively and correctly ranks completions which Sparrow does not. Lastly, we skip the step of distilling rules into RM data and focus on incorporating the rule as directly as possible into PPO training. Reinforcement Learning From AI Feedback (RLAIF) To address the cost and time of collecting human data, work that uses AI feedback to improve models have been a topic of recent study in both safety (such as CAI [ 10 , 11 ] ), and non-safety settings (RLAIF [ 9 ] ). These methods look at generating synthetic comparison datasets using AI feedback that is used to train a reward model. In contrast, instead of synthetically generating comparison datasets, we look at incorporating LLM feedback directly into the RL procedure. We additionally differ by using fine-grained and