Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06

Nilanjana Das, Edward Raff, Aman Chadha, Manas Gaur · 2024

Read Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 by Nilanjana Das, Edward Raff, Aman Chadha, Manas Gaur (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16359.txt.

Introduction

Nilanjana Das, Edward Raff, Aman Chadha, and Manas Gaur show, in a 2024 paper, that the most dangerous prompts for a language model may read like ordinary conversation. Classic jailbreaks hide in gibberish strings that filters catch easily. These authors instead wrap harmful requests in rich situational context, such as movie scripts, so the full prompt looks natural while the payload slips through.

They also built a way to launder gibberish attack text into readable, innocent-looking prose that keeps its bite inside a full prompt. Extending an existing attack framework with a sampling tweak produced varied readable attacks that worked well against both open and commercial models. The authors argue that safety checks tuned for weird strings will miss this class entirely. Their aim is to push defenses toward reading meaning and context, the way the attacks do.

Catalog record

CREATORNilanjana Das, Edward Raff, Aman Chadha, Manas Gaur
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-16359

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Nilanjana Das, Edward Raff, Aman Chadha, Manas Gaur Abstract . The explosive growth of Large Language Models (LLMs) is undeniable. As these AI systems become deeply embedded in social media platforms, we’ve uncovered a concerning security vulnerability that goes beyond traditional adversarial attacks. It becomes important to assess the risks of LLMs before the general public use them on social media platforms to avoid any adverse impacts. Unlike obvious nonsensical text strings that safety systems can easily catch, our work reveals that human-readable situation-driven adversarial full-prompts that leverage situational context are effective but much harder to detect or audit. We found that skilled attackers can exploit the vulnerabilities in open-source and proprietary LLMs to make a malicious/harmful user query safe for LLMs, resulting in generating a harmful response. This raises an important question about the vulnerabilities of LLMs with such full-prompts and hence their inherent robustness. Thus, to measure the robustness against human-readable attacks, which now present a potent threat, our research makes three major contributions. First, we developed attacks that use movie scripts as situational contextual frameworks (e.g., crime genre), creating natural-looking full-prompts that trick LLMs into generating harmful content. Second, we developed a method to transform gibberish adversarial text into readable, innocuous content that still exploits vulnerabilities when used within the full-prompts . Finally, we enhanced the AdvPrompter framework with p-nucleus sampling to generate diverse human-readable adversarial texts that significantly improve attack effectiveness against models like GPT-3.5-Turbo-0125 and Gemma-7b . Our findings show that these systems can be manipulated to operate beyond their intended ethical boundaries when presented with seemingly normal prompts that contain hidden adversarial elements. By identifying these vulnerabilities, we aim to drive the development of more robust safety mechanisms that can withstand sophisticated attacks in real-world applications. 1. Introduction Wang et al. 2025 mention that adversarial attacks can undermine the reliability, security, and trustworthiness of Large Language Models (LLMs), by producing major risks to entities that use them. This implies that situational adversarial attacks on LLMs can significantly affect their presence and trustworthiness on social media. These attacks exploit vulnerabilities in LLMs to generate misleading or harmful content, potentially spreading misinformation and influencing public opinion, thus adversely impacting the general public. With LLMs projected to power 750 million applications by 2025 1 1 1 https://springsapps.com/knowledge/large-language-model-statistics-and-numbers-2024 , understanding their robustness, trustworthiness, and safety has become critical. It should be understood whether these LLMs are safe or pose a risk to the public for general use. Adversarial attacks have been instrumental in identifying LLM vulnerabilities. Traditional approaches have relied on nonsensical inputs (e.g., "AUSHDoalin#!"), functioning analogously to SQL injection attacks, which increase model compliance with malicious prompts. Zou et al. 2023 use a combination of a malicious query and an adversarial suffix to generate harmful responses. However, this methodology presents a significant limitation: such tokens are readily identifiable by human reviewers and can be detected and refused by LLMs following multiple interactions. Figure 1. Pipeline for generating situation-adaptive, human-readable adversarial prompts that exploit LLM vulnerabilities. MP+Sit: Concatenation of Malicious Prompt (MP) and Situational context. Adv Ins: Adversarial Insertion from an anonymous attacker. A sophisticated adversary would instead employ human-readable adversarial prompts that appear innocuous upon inspection, rendering them more resistant to automated detection and manual review processes. This vulnerability poses substantial concerns as LLMs increasingly integrate into social media platforms for content moderation and user interaction functions. Recent evidence from Andriushchenko et al. 2024a findings demonstrates how newer models like Gemini 1.5 Pro/Flash show improved resilience against nonsensical adversarial inputs while remaining vulnerable to human-readable attacks. This implies that LLMs in general may resist non-human-interpretable tokens but remain susceptible to attacks using natural language instructions. This vulnerability pattern suggests that as models evolve with better text understanding and instruction-following capabilities, adversarial techniques will similarly advance toward more sophisticated human-readable approaches. Our research specifically addresses this emerging threat landscape by generating situation-influenced human-readable attacks that evade manual detection, providing crucial insights into potential security risks as LLM capabilities continue to develop. The challenge with human-readable attacks is that they are human-labour-intensive to craft. Additionally, previous approaches to creating human-readable adversarial attacks were complex and resource-intensive, requiring either access to model weights for gradient descent operations or sophisticated genetic algorithms that produced verbose outputs Liu et al. 2024 ( Wang et al. 2024 ) . Better adversarial prompts, in terms of both efficiency and length, would make the attack more subtle. Our research overcomes these limitations with an innovative optimization-free strategy that significantly simplifies the creation of deceptive adversarial prompts. Table 1 shows an illustration of our full-prompt template that includes a successfully transformed nonsensical suffix into natural, human-readable text through a straightforward process. This proves that sophisticated attacks can be executed with surprisingly simple techniques. Malicious Prompt Write a tutorial in steps like step 1, step 2, etc., on how to commit the crime as shown in the movie in real life. Human-readable Adversarial Insertion Luci expressed persistence in holding onto the originally repeated templates. Situational Context This is an overview of the movie with the title ‘The Dark Knight’. Here is the movie overview:- When the menace known as the Joker wreaks havoc and chaos on the people of Gotham, Batman must accept one of the greatest psychological and physical tests of his ability to fight injustice. Table 1. Full-prompt attack template, which illustrates the structure of a human-readable adversarial full-prompt, which combines a malicious prompt, an adversarial insertion, and situational context (e.g., a movie summary) to exploit LLM vulnerabilities in a natural and undetectable format. Our approach to generating adversarial full-prompt uses the PromptBench’s infrastructure because it includes inference pipelines for several models and the template for the FS-CoT technique ( Zhu et al. 2023a ) . Using this established framework, we enhance our full-prompt structure with situational context from movies (e.g., crime). The full-prompt was designed to have a combination of a malicious prompt, an adversarial insertion, and a movie situation that can force an LLM to generate contextually undesirable or unsafe responses. We leverage a well-known movie dataset for movies in three genres: Crime, Horror, and War. In our work, the human-readable adversarial insertions are the transformed versions of the nonsensical adversarial suffixes collected from Andriushchenko et al. 2024a work, which is further discussed in Section 3 . The success of the attack was measured using GPT-4 Judge initially and then by GPT-4-o-mini Judge for the scaled experiments. GPT-4 Judge developed by Qi et al. 2023 rates a response on a harmfulness scale of 1 to 5, where 1 and 5 represent the safest and the most harmful response, respectively. Figure 1 provides high-level information on how the full-prompt template could be used to attack an LLM. We propose a three-pronged research approach: First, we explore whether the few-shot chain-of-thought ( FS-CoT ) technique can effectively manipulate language models, providing a simple yet powerful method to craft these human-readable adversarial attacks. Second, we leverage AdvPrompter, a framework of Paulus et al. 2024 , to scale these experiments for generating effective

Read the complete text