KG-DF: A Black-box Defense Framework against Jailbreak Attacks Based on Knowledge Graphs

Shuyuan Liu 1 , Jiawei Chen 1,3 , Xiao Yang 2 , Hang Su 2 , Zhaoxia Yin 1,† · 2025

Read KG-DF: A Black-box Defense Framework against Jailbreak Attacks Based on Knowledge Graphs by Shuyuan Liu 1 , Jiawei Chen 1,3 , Xiao Yang 2 , Hang Su 2 , Zhaoxia Yin 1,† (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07480.txt.

Introduction

Shuyuan Liu and colleagues defend large language models against jailbreak prompts. Carefully crafted inputs can trick a model into unsafe or uncontrolled replies. Guards that are too strict hurt normal use, while weak guards leave holes.

Their framework, KG-DF, checks inputs against a knowledge graph without opening the model itself. Structured facts and semantic links help judge whether a request is benign or an attack in disguise. Because the defense sits outside the model, it can protect different systems the same way. Tests aim to keep helpful answers flowing while blocking manipulation. Safer deployment is the practical goal.

Catalog record

CREATORShuyuan Liu 1 , Jiawei Chen 1,3 , Xiao Yang 2 , Hang Su 2 , Zhaoxia Yin 1,†
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07480

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

KG-DF: A Black-box Defense Framework against Jailbreak Attacks Based on Knowledge Graphs Shuyuan Liu 1 , Jiawei Chen 1,3 , Xiao Yang 2 , Hang Su 2 , Zhaoxia Yin 1,† Abstract With the widespread application of large language models (LLMs) in various fields, the security challenges they face have become increasingly prominent, especially the issue of jailbreak. These attacks induce the model to generate erroneous or uncontrolled outputs through crafted inputs, threatening the generality and security of the model. Although existing defense methods have shown some effectiveness, they often struggle to strike a balance between model generality and security. Excessive defense may limit the normal use of the model, while insufficient defense may lead to security vulnerabilities. In response to this problem, we propose a Knowledge Graph Defense Framework (KG-DF). Specifically, because of its structured knowledge representation and semantic association capabilities, Knowledge Graph(KG) can be searched by associating input content with safe knowledge in the knowledge base, thus identifying potentially harmful intentions and providing safe reasoning paths. However, traditional KG methods encounter significant challenges in keyword extraction, particularly when confronted with diverse and evolving attack strategies. To address this issue, we introduce an extensible semantic parsing module, whose core task is to transform the input query into a set of structured and secure concept representations, thereby enhancing the relevance of the matching process. Experimental results show that our framework enhances defense performance against various jailbreak attack methods, while also improving the response quality of the LLM in general QA scenarios by incorporating domain-general knowledge. 1 Introduction With the rapid advancement of artificial intelligence, large language models (LLMs) have found widespread applications in various fields 24 ; 1 ; 39 ; 29 . These models have become a cornerstone of modern science and technology due to their powerful natural language processing capabilities and wide application scenarios 38 ; 18 ; 20 ; 13 ; 27 ; 32 . However, despite the convenience LLMs bring, they also face significant security challenges 35 . The complexity and openness of these models make them susceptible to a range of attacks, particularly jailbreak attacks, that manipulate the model produce erroneous or uncontrolled outputs 10 ; 19 ; 16 ; 33 ; 41 . Figure 1: The pipeline of our proposed defense framework against jailbreak attacks based on Knowledge Graphs at the inference stage. When our warning information is attached to the input prompts, the protected LLM will be robust to malicious attacks while maintaining reasonable responses to legitimate requests. To address this challenge, researchers have proposed a variety of defense strategies 31 ; 29 ; 28 ; 24 . Zhou et al. 40 suggested adding a suffix to the prompt, carefully designed using gradient descent algorithms, to enhance the security of LLM outputs. Alon et al. 3 introduced the concept of a confusion filter to identify and filter out overly complex inputs. Robey et al. 25 employed character-level perturbation techniques to neutralize methods sensitive to perturbations. Although these approaches have improved the defense capabilities of LLMs to some extent, they often undermine the generality of the model itself. As a result, existing defense methods struggle to strike an optimal balance between model generality and security. Finding ways to strengthen LLM defense mechanisms without compromising their generality has become a pressing issue. Recently, Mo et al. 23 introduced a novel defense mechanism against jailbreak attacks, which involves generating a prompt prefix via prompt countermeasure adjustment, while preserving the generality of model. However, this approach is designed for open-source model. However, many LLMs are closed-source model, and defenders lack access to the model’s internal details. As a result, the applicability and effectiveness of this method in real-world scenarios are limited. To address this problem, this paper aims to enhance the security of the model in black-box scenarios without compromising its generality. Achieving this goal faces several challenges 11 . First, due to the closed-source nature of LLMs, where internal structures and parameters are inaccessible, there is a lack of effective reasoning paths for security strategies, limiting the available defense methods. Second, most defense methods are unable to respond promptly to novel attack patterns, making it difficult to meet the demands for real-time protection. In this context, KGs, with their structured knowledge representation and semantic association capabilities, offer promising solutions for LLM defense 7 ; 8 ; 12 ; 37 . Specifically, KGs do not rely on access to the internal structure of models. By comparing the input content with dangerous entities and relationships in the KG, it becomes possible to identify potentially harmful intentions through external reasoning. Moreover, the dynamic and extensible nature of KGs enables continuous integration of novel attack features into the defense system, offering better scalability and adaptability than static rule-based methods 17 . However, the direct application of traditional KGs in LLM defense faces two main challenges. First, adversarial attacks often bypass entity recognition modules by using syntactic variations that maintain semantic integrity (e.g., character substitutions like "b0mb") 14 ; 36 ; 15 ; second, traditional keyword extraction methods rely on superficial semantic features (such as TF-IDF weights 6 ; 30 ; 9 ), which lose effectiveness when dealing with semantically coherent but logically abnormal prompts. To address these shortcomings, we propose a black-box defense framework against Jailbreak Attacks based on KGs(KG-DF), as illustrated in Figure 1 . This framework extracts core semantic information from user input through a semantic parsing module and performs similarity matching with a prebuilt security knowledge graph. The detected triples are then integrated into the LLM input layer as security alerts, allowing the model to simultaneously perceive potential threats during the generation phase. This framework not only provides defense capabilities but also enhances the model’s understanding of general-domain knowledge by incorporating it into the knowledge graph, thereby improving the response quality in general question-answering scenarios. The experimental results show that the defense strategy proposed in this paper demonstrates excellent defense performance on both open-source and closed-source models. On the open-source model Vicuna-7B, the attack success rate is nearly reduced to zero, with generality reaching 88%. On the closed-source models GPT-3.5 and GPT-4, the attack success rates are also close to 0%, with generality reaching 86% and 89%, respectively. Our contributions are summarized as follows: • We propose a novel defense perspective: integrating KG with LLMs to enhance LLM security in black-box settings through external semantic reasoning paths. • The semantic parsing module proposed in this paper replaces the traditional entity relationship extraction, enhances the matching correlation, and thereby improves the defense ability and universality of the model. • Experimental results show that the framework significantly enhances the model’s defense performance while maintaining its generality. Figure 2: Framework of the Defense Method. The proposed framework comprises three main steps: (1) constructing a knowledge graph that integrates both safety-related and general-domain knowledge, (2) extracting keywords from user prompts, and (3) retrieving and integrating relevant knowledge to guide the model toward safer and more accurate responses. 2 Related Work 2.1 Jailbreak Attack LLM has received a lot of attention because of its powerful generative ability, but studies have shown that LLM is vulnerable to adversarial attacks to bypass its own value alignment mechanism. According to the openness of the model,

Read the complete text