The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs

Sergey Berezin, Reza Farahbakhsh, Noel Crespi · 2025

Read The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs by Sergey Berezin, Reza Farahbakhsh, Noel Crespi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18626.txt.

Introduction

A hidden task tucked inside a prompt can trick a language model into saying what it should refuse. Sergey Berezin, Reza Farahbakhsh, and Noel Crespi name that trick in a 2025 paper. Their attacks hide banned requests inside chores like decoding a cipher, solving a riddle, or running code. The model does the chore and, in doing so, produces the banned content by a side door.

The team built a benchmark to grade these attacks across models. Six leading models were tested, among them GPT-4o and LLaMA 3.2. The attacks slipped past the guards in each case. The authors warn that current safety training has a blind spot here. They call for stronger defenses built for this attack class.

Catalog record

CREATORSergey Berezin, Reza Farahbakhsh, Noel Crespi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18626

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs Sergey Berezin, Reza Farahbakhsh, Noel Crespi Abstract We present a novel class of jailbreak adversarial attacks on LLMs, termed Task-in-Prompt (TIP) attacks. Our approach embeds sequence-to-sequence tasks (e.g., cipher decoding, riddles, code execution) into the model’s prompt to indirectly generate prohibited inputs. To systematically assess the effectiveness of these attacks, we introduce the PHRYGE benchmark. We demonstrate that our techniques successfully circumvent safeguards in six state-of-the-art language models, including GPT-4o and LLaMA 3.2. Our findings highlight critical weaknesses in current LLM safety alignment and underscore the urgent need for more sophisticated defence strategies. Warning: this paper contains examples of unethical inquiries used solely for research purposes. 1 Introduction 1.1 Background and Motivation Everything is a poison and a cure, and large language models (LLMs) are no exception to this principle. LLMs have achieved remarkable success in natural language processing, powering a wide range of applications including conversational agents, information retrieval systems, and automated content moderation tools. Due to their ability to generate human-like responses, LLMs are now extensively deployed across both consumer and enterprise sectors. However, widespread adoption also raises concerns about security, robustness, and misuse. Adversarial actors may exploit LLMs to spread toxic speech or even use these models as tools of crime Europol (2023) . Thus, ensuring that LLMs can withstand adversarial inputs and maintain reliable behaviour in sensitive scenarios is crucial. This paper introduces a novel class of attacks on LLMs, highlighting the urgent need for improved safeguards to preserve their intended use. 1.2 Existing Work and Problem Statement 1.2.1 Overview of Modern LLM Safeguard Mechanisms To prevent the generation of harmful, offensive, or otherwise restricted content, developers have introduced various safeguard mechanisms for large language models. These mechanisms generally aim to ensure that models behave within established ethical boundaries. Three common approaches include: • Filter-Based Approaches rely on predefined keyword filtering systems that block the generation of specific phrases, words, or concepts. Such systems may use extensive blacklists OpenAI (2023) or context-sensitive filtering techniques Varshney et al. (2023) ; Dong et al. (2024b) to identify and filter offensive content. • Reinforcement Learning with Human Feedback (RLHF) fine-tunes model weights according to human feedback on attributes such as helpfulness and harmlessness. This is typically done with a separate preference model, which is trained to predict which responses are more aligned with human preferences Dai et al. (2023) . • Neural-Symbolic Systems combine deep learning with symbolic reasoning components by applying rules to the outputs of neural networks. This could include counterfactual testing - making sure that the model provides the same answer after changing a demographic attribute in the input, semantic similarity checks and specific rule-based metrics. They integrate data-driven generation with strict rule enforcement for preventing hallucinations or mitigating biases in complex tasks Dong et al. (2024a) . 1.2.2 Adversarial Attacks on LLMs Adversarial attacks aim to circumvent these safeguards, prompting models to produce harmful, biased, or restricted content. Such attacks can exploit both the model’s input and its training data Shayegani et al. (2023) . Key types include: • Prompt-Based Attacks created by carefully crafting inputs to deceive the model into generating harmful content. Often this involves embedding indirect, ambiguous, or role-playing scenarios into the request Xu et al. (2023) ; Jiang et al. (2024) ; Shayegani et al. (2023) . Indirect prompt injection targeting retrieval-augmented models often exploits external knowledge sources or plug-in-based architectures. Such injections rely on tampering with external databases or documents to produce harmful content Yi et al. (2024) ; Ruck and Sutton (2024) . • Backdoor Attacks focus on inserting hidden triggers during the training process. By manipulating the training data, an adversary can create "backdoors" that remain dormant until specific inputs activate them. When triggered, these backdoors lead the model to generation of unsafe outputs Li et al. (2024) . Another vector of backdoor attacks is chain-of-thought prompting: inserting malicious reasoning steps causes unintended outputs when specific triggers are present Xiang et al. (2024) . • Perturbation attacks involve making slight, often imperceptible changes to the input, such as misspellings or syntactic changes, while preserving its semantic meaning. These small perturbations confuse the model’s internal mechanisms, leading it to generate outputs that deviate from intended behaviour Lin et al. (2024) . 1.3 Contribution The ArtPrompt jailbreak attack Jiang et al. (2024) introduced a method of bypassing LLM safety mechanisms by encoding keywords within prompts using ASCII art. In this attack, part of the input is encoded and the prompt provides explicit instructions on how the model should decode the hidden message. Using this approach, the authors successfully delivered malicious inputs to the LLM, circumventing its safeguards. Jiang et al. attributed the success of their attack to the inability of LLMs to interpret inputs spatially rather than semantically. However, with the guidance provided in the prompt, the model was able to accurately interpret the encoded input, revealing a contradiction in the spatial reasoning hypothesis. Recent work by Berezin et al. (2024) confirms that LLMs generally cannot interpret ASCII art as intended. This finding indicates that ArtPrompt’s mechanism does not rely on the ASCII art format itself. Notably, Jiang et al. carefully guided the model through a detailed decoding process, raising the question: Is ASCII art essential for the success of such attacks, or can any sequence-to-sequence (seq2seq) encoding task achieve similar results? Our investigation shows that the ASCII art format is not inherently necessary. We find that the ArtPrompt attack is a specific instance of a broader, previously unidentified class of vulnerabilities that we call Task-in-Prompt (TIP) attacks. TIP attacks embed a particular task, such as encoding, decoding, or problem-solving, directly into the prompt. By leveraging the model’s task-solving capabilities, attackers can indirectly produce restricted inputs, effectively circumventing safety constraints. Unlike prior jailbreaks that rely on isolated techniques (e.g., ASCII art in ArtPrompt or persona-based prompts in DAN Shen et al. (2024) ), TIP attacks highlight a broader vulnerability. By embedding illicit content within an arbitrary sequence-to-sequence task TIP attacks demonstrate that safe behaviour cannot be assured solely through the filtering of known patterns or formats. This observation emphasises that TIP attacks represent a general class of vulnerabilities, rather than isolated exploits. Our contributions are as follows: 1. We introduce TIP attacks as a previously unreported class of adversarial attacks. 2. We formally define TIP attacks and describe their key characteristics. 3. We develop a benchmark to evaluate TIP attacks across a range of contemporary LLMs. 4. We provide empirical evidence demonstrating that TIP attacks successfully bypass safety mechanisms in state-of-the-art LLMs. 2 Task-in-Prompt (TIP) Attack Figure 1: Overview of normal workflow (left) compared to the workflow with a Task-in-Prompt attack (right). A Task-in-Prompt (TIP) attack exploits the model’s core instruction-following capability, leveraging the longstanding challenge of separating instructions from data. LLMs, during safety alignment, learn to recognise and filter out certain trigger words or inquiries Wei et al. (2023) . By avoiding these specific trigger words and embedding unsafe content within a benign transformation task, an adversary could force the model to infer the word through an intermediate task, allowing them to circumvent detection. This suggests that as long as LLMs are designed to

Read the complete text