SecureBreak - A dataset towards safe and secure models

Marco Arazzi, Vignesh Kumar Kembu, Antonino Nocera · 2026

Read SecureBreak - A dataset towards safe and secure models by Marco Arazzi, Vignesh Kumar Kembu, Antonino Nocera (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21975.txt.

Introduction

Arazzi and co-authors build SecureBreak, a dataset for catching harmful outputs from large language models. Jailbreaks and prompt injection can slip past a model's built-in alignment, so a second filter layer is useful. SecureBreak labels model outputs across several risk categories, with careful manual labels set conservatively for safety. It serves two roles: feedback on alignment during training, and a final screen for deployed systems. Pretrained models improved at spotting unsafe content after fine-tuning on it. The authors offer it as shared ground for safety work.

Catalog record

CREATORMarco Arazzi, Vignesh Kumar Kembu, Antonino Nocera
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31180
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21975

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SecureBreak - A dataset towards safe and secure models Marco Arazzi, Vignesh Kumar Kembu, Antonino Nocera Abstract Large language models are becoming pervasive core components in many real-world applications. As a consequence, security alignment represents a critical requirement for their safe deployment. Although previous related works focused primarily on model architectures and alignment methodologies, these approaches alone cannot ensure the complete elimination of harmful generations. This concern is reinforced by the growing body of scientific literature showing that attacks, such as jailbreaking and prompt injection, can bypass existing security alignment mechanisms. As a consequence, additional security strategies are needed both to provide qualitative feedback on the robustness of the obtained security alignment at the training stage, and to create an “ultimate” defense layer to block unsafe outputs possibly produced by deployed models. To provide a contribution in this scenario, this paper introduces SecureBreak , a safety-oriented dataset designed to support the development of AI-driven solutions for detecting harmful LLM outputs caused by residual weaknesses in security alignment. The strong reliability of the proposed dataset derives from the adopted manual annotation procedure, in which labels are assigned conservatively to prioritize safety even in the presence of minor disagreements in the annotators’ opinion. Our exploratory data analysis campaign shows satisfactory performance in the detection of unsafe content across several risk categories. To evaluate its effectiveness, we measure the performance of several pre-trained LLMs in the considered classification setting under baseline conditions and compare these results with those obtained after fine-tuning the same models on SecureBreak . The results indicate that the dataset is valuable not only for constructing post-generation filtering modules that act as a last-line defense, but also for building additional supervisory intelligence for alignment optimization. In particular, classifiers derived from SecureBreak can be used to measure residual safety failures, inform whether additional training or refinement steps are necessary, and ultimately support more controlled and effective security alignment workflows. Content Warning: This paper contains examples of harmful language I Introduction Large Language Models (LLMs) have rapidly become integral part of a wide range of applications due to their ability to understand and generate human language. Advances in training data, model architectures, and computational power have enabled LLMs to perform complex tasks, such as information retrieval, decision support, coding assistance, and text analysis with increasing accuracy. LLMs have been successfully adopted even in critical domains such as healthcare, where they have been adopted to enhance clinical decision-making, automate administrative processes, and improve patient engagement through the analysis of records [ 1 ] . However, the use of this technology in high-risk application scenarios poses a relevant constraint in terms of security. In addition, recent research on LLM security has shown that carefully crafted malicious inputs can trick models into generating harmful contents, even if most of the existing commercial and open-source LLMs are originally designed to block them. In fact, safety in LLMs is based primarily on the internal alignment mechanisms of the model, which are designed to reduce the probability of generating unsafe content. However, these mechanisms are not flawless and can be bypassed using adversarial techniques. One of the most common and effective security threats against LLM is the jailbreaking attack, which involves crafting specific prompts designed to bypass the models safety mechanisms [ 2 ] . This motivates the introduction of additional independent defense layers based on an evaluation of the unsafety level of the produced output or post-generation filtering. This layered approach may serve a twofold objective: (i) it can act as an ultimate defense against unsafe outputs generated by alignment failures, and (ii) it can provide a supervisory signal that can be exploited to assess alignment quality, detect unresolved weaknesses, and guide subsequent security re-alignments. These independent additional defense layers can be built employing, once again, the power of Artificial Intelligence (AI, for short) solutions by adopting specifically crafted training datasets. However, most of the existing safety and alignment datasets for large language models are built around question‑level or input‑level harmfulness judgments, where the primary task is to classify individual prompts. Benchmarks such as the JailbreakBench dataset focus on harmful questions to evaluate the model refusal behavior, but they do not fully capture how harmful outputs may arise only in extended contexts or through subtle compositional interactions in input sequences [ 3 ] . To address the challenge above, in this paper, we introduce a dataset, called SecureBreak , specifically designed to classify safe and unsafe responses. This dataset enables the development of reliable response-level classifiers, which we have tested to confirm their effectiveness in detecting potentially harmful and unsafe output. The results of our experimental evaluation show how the proposed dataset is effective not only for the development of post-generation verification and filtering modules serving as an “ultimate” defensive layer, but also to derive supervisory components to improve the alignment quality of a target LLM. Link to the dataset - https://github.com/VIGNESH-KUMAR-KEMBU/SecureBreak II RELATED WORK Nowadays Large Language Models are widely used as assistant in many different tasks and by a large variety of people. So ensuring the robustness an Safetiness of the responses produced by these models is becoming a primary research objective [ 4 ] . To overcome this limit of big models trained on a massive amount of data without any restriction a strategy of Alignment of the model using Low Rank Adapters (LoRA) [ 5 ] can be used to introduce safeguards into the model without training or fine-tuning the base model. Low-Rank Adapters were introduced to enable efficient training of large models by updating only a small subset of parameters, making fine-tuning feasible even on less capable GPUs. This technology is then been exploited to efficiently align big models on specific domains [ 6 , 7 ] , introduce specific behaviors like avoid harmful responses [ 8 ] or even used to infer information from the base model [ 9 ] . With this objective, datasets like HH-RLHF [ 10 ] , BeaverTails [ 11 ] or Do Not Answer [ 12 ] have been proposed as baselines for the training of safety-focused alignment adapters. Despite these safeguards, there are adversarial attacks that can bypass such restrictions. As previously observed in other scenarios, where backdoor attacks can alter a model’s behavior through small manipulation to the input [ 13 , 14 , 15 ] , jailbreak attacks operate in a similar fashion by modifying the prompt to circumvent the safety constraints [ 16 , 17 , 18 , 19 , 20 ] . To this purpose datasets like AdvBench [ 21 ] and JailbreakBench [ 3 ] serve as benchmark to test models on adversarial prompts. In AdvBench [ 21 ] , the authors produced harmful prompts by using an uncensored Vicuna model, asking it to generate new strings based on five demonstration examples that they had written themselves. In JailbreakBench [ 3 ] , instead the authors provide an evolving repository of artifacts corresponding to state-of-the-art jailbreaking attacks and defenses. SecureBreak distinguishes itself by shifting focus from prompt analysis to response-level classification specifically within adversarial jailbreak contexts. Unlike benchmarks relying on automated evaluators, our dataset leverages high-quality human annotation to capture subtle safety violations. This design is expressly intended to facilitate

Read the complete text