LeakSealer: A Semisupervised Defense for LLMs Against Prompt Injection and Leakage Attacks

Francesco Panebianco, Stefano Bonfanti, Francesco Trovò, Michele Carminati · 2025

Read LeakSealer: A Semisupervised Defense for LLMs Against Prompt Injection and Leakage Attacks by Francesco Panebianco, Stefano Bonfanti, Francesco Trovò, Michele Carminati (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00602.txt.

Introduction

Francesco Panebianco, Stefano Bonfanti, Francesco Trovò and Michele Carminati present LeakSealer, a semi-supervised defence for large language models against prompt injection and data leakage. Wider use of these models, plus retrieval systems that feed them private text, has opened paths for jailbreaks and leaks of sensitive facts. Their work has two parts, one forensic and one active.

Past chat logs are first mapped by topic, which shows how attack patterns shift over time. The live defence then groups topics, flags odd patterns, and brings a human into the loop. On a public toxic chat set it led on precision and recall for injection spotting. For private data leaks it reached a strong score well above the baselines.

Catalog record

CREATORFrancesco Panebianco, Stefano Bonfanti, Francesco Trovò, Michele Carminati
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00602

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LeakSealer: A Semisupervised Defense for LLMs Against Prompt Injection and Leakage Attacks Francesco Panebianco, Stefano Bonfanti, Francesco Trovò, Michele Carminati Abstract The generalization capabilities of Large Language Models (LLMs) have led to their widespread deployment across various applications. However, this increased adoption has introduced several security threats, notably in the forms of jailbreaking and data leakage attacks. Additionally, Retrieval Augmented Generation (RAG), while enhancing context-awareness in LLM responses, has inadvertently introduced vulnerabilities that can result in the leakage of sensitive information. Our contributions are twofold. First, we introduce a methodology to analyze historical interaction data from an LLM system, enabling the generation of usage maps categorized by topics (including adversarial interactions). This approach further provides forensic insights for tracking the evolution of jailbreaking attack patterns. Second, we propose LeakSealer, a model-agnostic framework that combines static analysis for forensic insights with dynamic defenses in a Human-In-The-Loop (HITL) pipeline. This technique identifies topic groups and detects anomalous patterns, allowing for proactive defense mechanisms. We empirically evaluate LeakSealer under two scenarios: (1) jailbreak attempts, employing a public benchmark dataset, and (2) PII leakage, supported by a curated dataset of labeled LLM interactions. In the static setting, LeakSealer achieves the highest precision and recall on the ToxicChat dataset when identifying prompt injection. In the dynamic setting, PII leakage detection achieves an AUPRC of 0.97 0.97 , significantly outperforming baselines such as Llama Guard. K eywords Prompt Injection \cdot Privacy \cdot Data Leakage \cdot Defense \cdot Jailbreaking \cdot Large Language Models 1 Introduction Large Language Models (LLMs) have revolutionized numerous tasks due to their emergent capabilities, establishing them as essential components across various applications yang_harnessing_2024 . LLMs are particularly powerful as question-answering systems, especially when integrated in a Retrieval-Augmented Generation (RAG) gao_retrieval-augmented_2023 pipeline, which incorporates relevant documents into the query context. RAG integrates context-dependent information, leading to more comprehensive and accurate responses. However, retrieved documents may contain classified information such as Personally Identifiable Information (PII). This sensitive data, if compromised, could be exploited by malicious actors for personal gain, such as through unauthorized disclosure or sale Sancho2023 . LLMs are susceptible to Prompt Injection, which can cause the model to deviate from its intended functionality, potentially producing toxic or harmful outputs (LLM Jailbreaking) shen__2023 or leaking sensitive data zeng_good_2024 . Existing solutions, such as Llama Guard inan_llama_2023 and LLM-As-A-Judge huang_empirical_2024 ; zheng_judging_2024 , rely on LLM-based approaches, requiring costly inference for each new sample. Thus, a service provider would perform twice the number of inference calls to the model to safeguard it against threats. The detection of past attacks can provide critical insights, enabling the identification of known adversaries. This process reveals patterns for active prevention of future attacks. The final output is a detailed usage map of a service, including trends and usage patterns that suggest product improvements. Original Contributions. The main contributions of our work are: • A general approach to perform historical analysis on interactions with an LLM. The method provides a usage map and forensic insight from the system conversation history; • LeakSealer, a lightweight semipervised framework for analyzing LLM systems under static and dynamic conditions. In the static scenario, it minimizes human effort in the historical analysis. In the dynamic approach, it serves as an active defense; • We produce a diverse dataset of labeled LLM conversations to detect leakage of PII from an RAG context. • We open-source both LeakSealer and the PII dataset to allow reproducibility and foster future research. More specifically, we introduce an approach to perform historical analysis on a corpus of interactions with an LLM. The method identifies semantic fingerprints of interactions with the LLM and groups matching samples for easier human inspection. The final output reports a comprehensive usage map that highlights topics of interest in the real-world application of an LLM system. For instance, the map can reveal whether a group of users is using a platform designed for AI-generated cooking recipes to suggest medications. The method allows the service provider to adjust the system’s design to accommodate spontaneous usage and provides forensic insight by grouping interactions with similar attack patterns and tagging them with characterizing keywords (e.g., topics outside the platform guidelines or attempts to leak phone numbers). Additionally, repeated analysis can offer temporal tracking of jailbreak attack patterns, facilitating monitoring of their evolution. Our second contribution is LeakSealer , a model-agnostic framework to address prompt injection attacks on standalone and RAG-based systems. The framework consists of two approaches that seamlessly blend into one another. The first is a static approach, applicable to an already deployed system. It implements the historical analysis methodology through an insightful usage of clustering and text analysis. The second is a dynamic approach, which leverages actionable patterns identified in the static approach to implement active defenses and counter ongoing attacks. Effective filtering is achieved in this scenario through a Human-In-The-Loop (HITL) pipeline. This allows for further specialization of the model in correctly identifying private PII versus publicly available PII. LeakSealer employs computationally efficient procedures, facilitating more cost-effective, rapid, and efficient detection compared to existing baselines. LeakSealer’s source code will be publicly available to support reproducibility and future development. We assess the performance of LeakSealer under two adversarial scenarios: (1) attempts by an attacker to jailbreak the LLM to generate unauthorized content and (2) targeted leakage of PII. For the first scenario, experiments are conducted using toxic interaction benchmarks commonly employed in the literature (OpenAI Content Moderation Dataset openai_2023_holistic and ToxicChat lin2023toxicchat ). For the second scenario, we address the absence of a high-quality dataset for defending against PII leakage by curating a dataset consisting of labeled interactions with an LLM. We will open-source the dataset as a benchmark for future research in this domain. In the static setting, LeakSealer performs exceptionally well in recognizing groups of interactions affected by prompt injection. The purity of identified clusters on the ToxicChat evaluation set reaches 0.97 , and its F1-score on both the ToxicChat and the PII dataset is the largest. LeakSealer also leads in performance in the dynamic setting. The Area Under the Precision-Recall Curve (AUPRC) is 0.97 , while the second-best baseline, Llama Guard, only reaches 0.84 . 2 Background on Security of LLMs This section introduces the background concepts of prompt injection and PII leakage in the context of LLMs. To the interested reader, we deferred to Appendix E the basic background on LLMs. Prompt Injection. Prompt Injection is an attack on LLMs, which allows malicious users to influence the model’s behavior, leading it to operate outside its predefined parameters shen__2023 ; yu_dont_2024 ; inan_llama_2023 ; xu_llm_2024 ; deng_masterkey_2024 . As is common in information security, the vulnerability arises from improper separation between instructions and data (user input). This allows malicious user inputs to modify or extend the model’s system prompt. One of the first instances of this attack is doing anything now (DAN), a roleplaying prompt devised by the jailbreak community to bypass ChatGPT restrictions shen__2023 . Jailbreak prompts employ various strategies to bypass content restrictions, often misleading the model with seemingly benign requests. Yu et al. yu_dont_2024 identify four categories of jailbreak prompts. The first is Disguised Intent , where the prompt frames a harmful request as a research inquiry or joke. The second is

Read the complete text