Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning

Zhaoqi Wang 1 , Daqing He 1 , Zijian Zhang 1 , Xin Li 1 , Liehuang Zhu 1 , Meng Li 2 , Jiamou Liu 3 · 2025

Read Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning by Zhaoqi Wang 1 , Daqing He 1 , Zijian Zhang 1 , Xin Li 1 , Liehuang Zhu 1 , Meng Li 2 , Jiamou Liu 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23558.txt.

Introduction

Attackers craft prompts that trick language models into breaking their own rules, a practice called jailbreaking. Zhaoqi Wang and 6 co-authors propose PASS, an attack that rewrites such prompts in formal dress. Reinforcement learning reshapes seed prompts into semantic and structural formalizations, which look harmless and slip past alignment defenses. The outputs then feed a GraphRAG system that reuses extracted terms and symbols as context.

That stored context strengthens later rounds of attack. Tests on common open-source models showed the method works well at eliciting banned responses. The authors frame the work as a way to expose weak spots in current alignment methods.

Catalog record

CREATORZhaoqi Wang 1 , Daqing He 1 , Zijian Zhang 1 , Xin Li 1 , Liehuang Zhu 1 , Meng Li 2 , Jiamou Liu 3
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning Zhaoqi Wang 1 , Daqing He 1 , Zijian Zhang 1 , Xin Li 1 , Liehuang Zhu 1 , Meng Li 2 , Jiamou Liu 3 Abstract Large language models (LLMs) have demonstrated remarkable capabilities, yet they also introduce novel security challenges. For instance, prompt jailbreaking attacks involve adversaries crafting sophisticated prompts to elicit responses from LLMs that deviate from human values. To uncover vulnerabilities in LLM alignment methods, we propose the PASS framework ( P rompt J a ilbreaking via S emantic and S tructural Formalization). Specifically, PASS employs reinforcement learning to transform initial jailbreak prompts into formalized descriptions, which enhances stealthiness and enables bypassing existing alignment defenses. The jailbreak outputs are then structured into a GraphRAG system that, by leveraging extracted relevant terms and formalized symbols as contextual input alongside the original query, strengthens subsequent attacks and facilitates more effective jailbreaks. We conducted extensive experiments on common open-source models, demonstrating the effectiveness of our attack. Content warning: This paper contains unfiltered LLM outputs that may be offensive. I Introduction Large language models (LLMs) such as GPT-4 [ 1 ] , LLaMA-3 [ 2 ] , and DeepSeek [ 3 ] have demonstrated superior capabilities in understanding, reasoning, and generation across various Natural Language Processing (NLP) tasks. This has led to their widespread application in tasks such as dialogue systems, text generation, and code generation. However, they also introduce new security risks. LLMs acquire knowledge from their training corpora and generate outputs based on inputs. This process can lead to LLMs producing responses that do not align with human values, such as content related to gore, violence, or illegal activities. Consequently, enabling LLMs to identify malicious intent and generate responses that align with human values has emerged as a pressing issue. To address this pressing issue, alignment techniques have emerged, which aim to ensure that LLMs’ outputs are consistent with human values and intentions. Among these, Reinforcement Learning from Human Feedback (RLHF) [ 4 , 5 ] has proven to be a groundbreaking technique for aligning LLMs. Following the introduction of RLHF, numerous studies have explored various approaches to further align LLMs. However, concurrently, an increasing number of jailbreak attack methods have been proposed. The focus of these attacks has shifted from initially inducing LLMs to output jailbroken content with the highest probability [ 6 , 7 ] to increasing stealthiness to bypass alignment mechanisms [ 8 , 9 ] . Despite the existence of numerous jailbreak attack methods in prior work, most approaches rely on predefined templates or fixed strategies to construct jailbreak prompts. This characteristic allows defenders to rapidly patch these vulnerabilities by augmenting their malicious input corpora, thereby rendering the attack methods ineffective. Even seemingly adaptive approaches like RL-JACK [ 10 ] , which uses reinforcement learning to automate prompt construction, exhibit critical flaws. Its reliance on basic rewriting actions like crossover and rephrasing is insufficient to effectively conceal malicious intent. Furthermore, its reward mechanism’s dependency on an unaligned model to calculate cosine similarity weakens the attack’s premise, as the desired harmful content is already obtainable. To address these limitations, we propose PASS ( P rompt J a ilbreaking via S emantic and S tructural Formalization). Our work introduces a novel attack approach that, crucially, does not depend on an external unaligned model. Instead, we formalize the structure and semantics of jailbreak prompts, decomposing the attack process into atomic, combinable steps. An RL agent is then trained to learn the optimal sequence of these steps, allowing for flexible and diverse formalization attack paths even for the same malicious query. Figure 1 shows, this method enhances the stealthiness needed to bypass modern alignment defenses. Furthermore, we introduce a novel mechanism for continuous learning by constructing a GraphRAG [ 11 ] system. This system extracts formalized knowledge from successful attacks into a graph, allowing the agent to efficiently retrieve and reuse proven tactics to accelerate subsequent attacks. The graph structure is a natural fit, as the formalized knowledge, with its inherent entities and intricate relationships, can be seamlessly represented as nodes and edges. Fig. 1: A simplified example of the PASS attack, illustrating the initial round’s malicious input and LLM response, alongside the final successful jailbreak input and its corresponding LLM response. In summary, our contributions are as follows: • We propose a novel jailbreaking attack method, named PASS, based on the formalization of jailbreak prompts. Our method employs reinforcement learning to achieve multi-round jailbreaking. To the best of our knowledge, this is the first work to guide attacks using formalized prompt descriptions and to extract this formalized knowledge for constructing a GraphRAG system. • We conduct extensive experiments to evaluate the effectiveness and practicality of PASS. Evaluation results against baselines confirm its high effectiveness and practical applicability in achieving stealthy and effective jailbreaks against aligned LLMs. • We formally analyze the underlying reasons for the attack’s success, revealing how our proposed method exploits the inherent limitations and vulnerabilities of current alignment mechanisms in LLMs. II Related Work The rapid development of LLMs has led to significant advancements in various NLP tasks. However, concerns regarding their safety and alignment with human values have also emerged, prompting extensive research into alignment techniques. The emergence of Retrieval Augmented Generation (RAG) has, to some extent, addressed the hallucination problem by introducing external context, with frameworks like GraphRAG [ 11 ] further enhancing this by integrating knowledge graphs to leverage structured knowledge for more precise and contextually rich information. The training corpora for LLMs are frequently derived from extensive web-scraped data. Consequently, their deployment can lead to behaviors that conflict with widely accepted norms, ethical standards, and regulations. To mitigate these issues, a substantial body of research has focused on aligning LLMs with human values and intentions. Specifically, RLHF was introduced as a pivotal technique to fine-tune language models using human feedback, thereby aligning their behavior with user intent across a broad spectrum of tasks [ 4 , 5 ] . Subsequently, Reinforcement Learning from AI Feedback (RLAIF) was explored as an alternative to human supervision [ 12 ] . More recently, Direct Preference Optimization (DPO) was proposed, which streamlines the RLHF training paradigm by directly optimizing a policy from preferences, obviating the need for an explicit reward model [ 13 ] . Building upon DPO, Online DPO was introduced, facilitating continuous refinement of alignment policies [ 14 ] . Meanwhile, an increasing number of adversarial attack methods have been proposed to bypass these alignment efforts. For instance, a method combining greedy and gradient-based discrete optimization was introduced to compute and append an adversarial suffix to harmful instructions, thereby automating the generation of jailbreak prompts without requiring manual crafting for each instance [ 6 ] . However, the prompts generated by this approach often contain a significant amount of garbled characters, making them susceptible to detection by perplexity-based defense mechanisms [ 15 ] . Building on these foundations, other works have explored diverse strategies for crafting adversarial prompts. A genetic algorithm-based search is leveraged to iteratively generate and refine jailbreak prompts [ 7 ] . Iterative semantic prompt optimization techniques are introduced to enhance both attack success rates and transferability across models [

Read the complete text