Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models
Neel Jain, Aditya Shrivastava, Chenyang Zhu, Daben Liu, Alfy Samuel, Ashwinee Panda, Anoop Kumar, Micah Goldblum, Tom Goldstein · 2024
Read Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models by Neel Jain, Aditya Shrivastava, Chenyang Zhu, Daben Liu, Alfy Samuel, Ashwinee Panda, Anoop Kumar, Micah Goldblum, Tom Goldstein (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06748.txt.
Introduction
Neel Jain, Aditya Shrivastava, Chenyang Zhu, and co-authors tackle a practical safety problem in this 2024 paper. Large language models must refuse some requests, such as instructions for illegal acts or questions past their knowledge. Yet different users want different refusal rates, and training a fresh model for each setting is costly.
The team adds special refusal tokens during training, one for each refusal category or a single shared token. At run time, raising or lowering the chance of that token steers how often the model refuses, category by category. One trained model can thus be tuned on the fly, with no extra fine-tuning. The approach gives builders a cheap dial for matching a model's caution to each user's needs.
Catalog record
| CREATOR | Neel Jain, Aditya Shrivastava, Chenyang Zhu, Daben Liu, Alfy Samuel, Ashwinee Panda, Anoop Kumar, Micah Goldblum, Tom Goldstein |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06748
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models Neel Jain, Aditya Shrivastava, Chenyang Zhu, Daben Liu, Alfy Samuel, Ashwinee Panda, Anoop Kumar, Micah Goldblum, Tom Goldstein Abstract A key component of building safe and reliable language models is enabling the models to appropriately refuse to follow certain instructions or answer certain questions. We may want models to output refusal messages for various categories of user queries, for example, ill-posed questions, instructions for committing illegal acts, or queries which require information beyond the model’s knowledge horizon. Engineering models that refuse to answer such questions is complicated by the fact that an individual may want their model to exhibit varying levels of sensitivity for refusing queries of various categories, and different users may want different refusal rates. The current default approach involves training multiple models with varying proportions of refusal messages from each category to achieve the desired refusal rates, which is computationally expensive and may require training a new model to accommodate each user’s desired preference over refusal rates. To address these challenges, we propose refusal tokens, one such token for each refusal category or a single refusal token, which are prepended to the model’s responses during training. We then show how to increase or decrease the probability of generating the refusal token for each category during inference to steer the model’s refusal behavior. Refusal tokens enable controlling a single model’s refusal rates without the need of any further fine-tuning, but only by selectively intervening during generation. 0 0 footnotetext: † Work completed during Capital One Internship; Correspondence to < < [email protected] > > 1 Introduction An essential property of a useful language model is the ability to produce refusal messages at appropriate times. Refusal messages enhance not only the safety of LLMs but also their utility and trustworthiness, as refusal messages can prevent LLMs from hallucinating or answering invalid requests. For example, an LLM that lacks the ability to browse the web should refuse when asked to access and summarize the content behind a URL. Likewise, a model should provide an informative refusal when asked to answer a question that is too under-specified or poorly formed to be answerable. To minimize hallucinations and unsafe behavior, instruction models like GPT-4 ( Achiam et al. 2023 ) and llama-3 ( Dubey et al. 2024 ) have been processed with alignment pipelines that imbue them with extensive refusal capabilities. Despite advancements in model finetuning and alignment, controlling refusal messages in these models remains a challenging task. For instance, llama-2-Chat ( Touvron et al. 2023 ) experienced issues with over-refusal, where the model would refuse too many queries, negatively impacting usability, mostly likely due to a post-training set with too many refusal messages. Simple approaches, such as training multiple models with varying levels of refusal data until the desired rates are achieved ( Dubey et al. 2024 ) are resource-intensive and still lack the precision to carefully adjust different categories of refusals. Moreover, the criteria for refusing is constantly evolving. What is considered an acceptable refusal for one use case or time may not align with the ethical, legal, or technical standards in a different setting. To address these weaknesses, we introduce a simple strategy that makes refusal behavior controllable at test-time without retraining: the refusal token. During alignment, we prepend a special [refuse] token to responses that contain a refusal. The model quickly learns to generate this token before refusing, and then to refuse when this token is present. At test-time, the softmax probability of the refusal token can be used as a metric for how likely it is that a refusal is necessary. By thresholding on this probability, one can turn a knob to control the refusal sensitivity after the model is trained. By employing different refusal tokens for different refusal types, one can impose fine-grained control over refusal behavior along different axes of behavior, and carefully optimize refusal rates in this multi-dimensional space. Our main contributions are the following: • We introduce a refusal token strategy. By thresholding the probability of this refusal token, we give model developers calibrated control over refusal rates without retraining. This development opens the door for sophisticated post-training calibration of refusal rates. For example, with minimal computation, one could sweep over refusal thresholds and select a value that achieves a specified rate of false refusals, or a value that maximizes an F1 score. • We show that multiple refusal tokens can manage different refusal message sets, enabling independent control over each refusal distribution. Additionally, we explore various strategies for manipulating these category-specific refusal tokens to meet test-time requirements. • We observe that the refusal token improves F1 scores, even without calibration. Furthermore, we highlight the importance of reducing Type II errors by including contrast or borderline examples in the training data. These examples, which are similar to refusal queries but innocuous, help refine the token’s effectiveness—specifically, its ability to appropriately switch between refusal and response based on the corresponding meta-token. 2 Related Work Refusal messages. The ability of generative models to refuse certain messages is particularly crucial for mitigating toxicity and reducing hallucinations. In the context of toxicity, several studies explore how language models respond to toxic prompts or instructions. One popular approach is to train an external model to determine whether the model should reject or respond to queries ( Dubey et al. 2024 ) . Bianchi et al. 2024 demonstrate that incorporating refusals into training data does not diminish a model’s helpfulness but can lead to over-refusals, where the model declines to respond even on innocuous requests. Similarly, Cui et al. 2024 ; An et al. 2024 investigate over-refusal behavior across various language models, developing an evaluation framework to assess over-refusals in response to harmful prompts. Regarding hallucinations, Zhang et al. 2024 introduce an algorithm called R-Tuning, which prompts the model to state “I am unsure” or “I am sure” after a question and answer session, framing the problem as a discrimination task. Additionally, Kang et al. 2024 and Kapoor et al. 2024 propose alternative algorithms for alleviating the hallucination problem, focusing on instances where it is unclear whether the model possesses the required knowledge. Feng et al. 2024 use multiple agents to determine when to abstain from queries. For predetermined queries the model is designed to refuse, Brahman et al. 2024 present a comprehensive taxonomy of such questions, highlighting scenarios where the model should appropriately refuse to respond. This work also releases instructional data designed to train models in this regard. Evaluative studies by Liu et al. 2023 , Yin et al. 2023 , and Amayuelas et al. 2024 further explore the types of questions that warrant refusal. Arditi et al. 2024 find a one-dimensional subspace such that erasing this specific direction from the model’s residual stream activations causes the model to consistently answer harmful queries. Concurrent work, Lee et al. 2024 extend Arditi et al. 2024 for usability, allowing additional steerability using activation steering to control refusal messages at test-time. Tagging, control codes, and meta-tokens. Sennrich et al. 2016 introduce a meta-token for machine translation, while Keskar et al. 2019 extend this idea by introducing control codes for more general control.. A control code is a piece of