Knowledge Graph Guided Evaluation of Abstention Techniques

Kinshuk Vasisht, Navreet Kaur, Danish Pruthi · 2024

Read Knowledge Graph Guided Evaluation of Abstention Techniques by Kinshuk Vasisht, Navreet Kaur, Danish Pruthi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07430.txt.

Introduction

Kinshuk Vasisht, Navreet Kaur, and Danish Pruthi test how well language models refuse harmless requests after safety training, in a 2024 paper. Safe deployment needs models that decline bad requests. But refusal training can overshoot, blocking innocent topics near the banned ones. To measure that spill, the authors built a benchmark from a knowledge graph of everyday concepts such as rivers.

Their benchmark, SELECT, asks models about safe concepts and their narrower subtopics. Six models, both open and commercial, faced several refusal methods. The methods blocked over eighty percent of target items, yet blocking fell by nineteen points on subtopics of those targets. No method won on all fronts; each traded breadth against precision differently. The findings help teams pick a refusal method with open eyes about its blind spots.

Catalog record

CREATORKinshuk Vasisht, Navreet Kaur, Danish Pruthi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-07430

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Knowledge Graph Guided Evaluation of Abstention Techniques Kinshuk Vasisht, Navreet Kaur, Danish Pruthi Abstract To deploy language models safely, it is crucial that they abstain from responding to inappropriate requests. Several prior studies test the safety promises of models based on their effectiveness in blocking malicious requests. In this work, we focus on evaluating the underlying techniques that cause models to abstain. We create SELECT , a benchmark derived from a set of benign concepts (e.g., “rivers”) from a knowledge graph. Focusing on benign concepts isolates the effect of safety training, and grounding these concepts in a knowledge graph allows us to study the generalization and specificity of abstention techniques. Using SELECT , we benchmark different abstention techniques over six open-weight and closed-source models. We find that the examined techniques indeed cause models to abstain with over 80 % 80\% abstention rates. However, these techniques are not as effective for descendants of the target concepts, where abstention rates drop by 19 % 19\% . We also characterize the generalization-specificity tradeoffs for different techniques. Overall, no single technique is invariably better than others, and our findings inform practitioners of the various trade-offs involved. 1 1 1 The benchmark and code to reproduce the evaluation with SELECT is available at https://github.com/kinshuk-h/SELECT 1 Introduction For several reasons, it is critical that language models abstain from responding to inappropriate user requests. These requests could include (malicious) attempts to assist users in illegal activity ( Fang et al., 2024 ; Weidinger et al., 2021 ) , generate offensive content ( Deshpande et al., 2023 ; Gehman et al., 2020 ) , or disseminate large-scale misinformation ( Tamkin et al., 2021 ; Buchanan et al., 2021 ) . To block such requests, a common approach is to perform some form of “safety” post-training before releasing language models ( Bai et al., 2022 ) . Safety post-training methods encompass supervised fine-tuning (SFT) ( Zhou et al., 2023 ) and reinforcement learning ( Ouyang et al., 2022 ; Rafailov et al., 2023 ) to align model outputs with human preferences. Figure 1: Leveraging knowledge graphs to evaluate abstention techniques. Ideally, abstaining from a concept should imply abstention for descendants (generalization) but not ancestor or sibling concepts (specificity). Several recent benchmarks evaluate the efficacy of safety-trained (or “aligned”) models, with some focusing on specific facets of safety. For instance, SORRY-Bench ( Xie et al., 2024 ) and OR-Bench ( Cui et al., 2024 ) evaluate models’ capabilities to refrain from answering queries related to certain unsafe and toxic topics, respectively. Similarly, other studies assess whether models abstain from sharing information about protected groups ( Chen et al., 2023 ) or answering questions outside their parametric knowledge ( Liu et al., 2024 ) . While existing benchmarks provide insights into how well current models safeguard against inappropriate requests, there is relatively little work that attempts to benchmark the underlying techniques used to enforce (or encourage) abstention in models. In this work, we introduce SELECT ( S elective-abstention E valuated by L everaging an E xtensive C oncept T axonomy), a benchmark to evaluate the effectiveness, generalization and specificity of abstention techniques. SELECT derives benign concepts from a knowledge graph (KG) ( Suchanek et al., 2024 ) . By focusing on benign concepts, we intend to isolate the effects of safety-training approaches performed for sensitive topics. Further, the connection with KGs allows us to exploit the hierarchical relations between entities to evaluate generalization and specificity. For concepts—or composition of concepts—in SELECT , we examine several abstention techniques and assess their: (a) effectiveness : whether models abstain from responding to questions about a given concept; (b) generalizability : whether abstention extends to concept’s descendants, and (c) specificity : whether questions about ancestor and sibling concepts are not refused, where abstention is undesirable. As illustrated in Figure 1 , for a given concept of ‘rivers’, the desirable outcome of an abstention technique would be to abstain from queries about rivers, generalizing to queries about specific rivers (e.g. ‘Nile’), while answering queries about ‘oceans’ (sibling) or other ‘water bodies’ (parent). Similarly, for a composition of concepts such as ‘water bodies in England’, abstention should generalize to ‘rivers in London’ and other ‘water bodies in England’, but not to ‘water bodies in Africa’, ‘rivers in Egypt’, or ‘England’ in general. Using SELECT , we evaluate abstention techniques including prompting ( Zheng et al., 2024 ) , activation steering ( Lee et al., 2024 ) and fine-tuning ( Bianchi et al., 2024 ; Rafailov et al., 2023 ) for both closed-source and open-weight models, including GPT-4o ( OpenAI, 2024 ) and LLaMA-3.1 ( Dubey et al., 2024 ) . Through our evaluation, we find that abstention techniques are effective for abstention from benign concepts and their compositions, measured using abstention rates being over 80\% for most techniques and further reaching close to 100\% in some cases. However, the abstention effects do not necessarily extend to descendants, where we observe abstention rates drop by 19\% on average. Interestingly, different techniques trade-off between generalization and specificity in varied ways. Most techniques perform similarly in terms of effectiveness corresponding to concepts at varying depths in the taxonomy. Somewhat surprisingly, specificity decreases for concepts that are farther from the root, suggesting that abstention for narrower concepts leads to over-refusal. Overall, no single abstention technique is invariably better across all aspects. Our analysis finds prompting and activation steering to be more effective than fine-tuning approaches in causing the models to abstain for a given concept, as per higher abstention rates (+ 4 - 19 %) on average. However, the effectiveness for these techniques drops by 21 - 26\% for descendants, compared to 7\% for fine-tuning using SFT. Prompting and activation steering also result in higher gaps between generalization and specificity ( 26 - 47\% ) compared to fine-tuning ( 10\% ), indicating that prompting and activation steering lead to over-refusals. We believe our findings shed light into the strengths and weaknesses of different abstention techniques, and highlight the various tradeoffs at play. Additionally, we hope our evaluation informs practitioners about adapting models to novel but sensitive topics. 2 Related Work Evaluating Abstention in Language Models. When deploying large language models, it is crucial to abstain from inappropriate or malicious user requests ( Weidinger et al., 2021 ; Fang et al., 2024 ) . User requests—and corresponding model responses—may lead to bias, discrimination or toxicity ( Deshpande et al., 2023 ; Gehman et al., 2020 ) . Further, users may request information that is not captured within the parametric knowledge of the model ( Feng et al., 2024 ) , responses to which may be used to disseminate misinformation ( Tamkin et al., 2021 ; Buchanan et al., 2021 ) . Several recent efforts benchmark different facets of abstention in language models. For instance, SORRY-Bench ( Xie et al., 2024 ) focuses on benchmarking the ability of models to abstain from user requests considered harmful and malicious. The study benchmarks various open and closed source models, finding closed-source models display tolerable refusal rates across different categories of harm. Further, UnknownBench ( Liu et al., 2024 ) evaluates the ability of models to abstain in scenarios where models do not possess adequate knowledge

Read the complete text