Lightweight Safety Guardrails via Synthetic Data and RL-guided Adversarial Training
Aleksei Ilin, Gor Matevosyan, Xueying Ma, Vladimir Eremin, Suhaa Dada, Muqun Li, Riyaaz Shaik, Haluk Noyan Tokgozoglu · 2025
Read Lightweight Safety Guardrails via Synthetic Data and RL-guided Adversarial Training by Aleksei Ilin, Gor Matevosyan, Xueying Ma, Vladimir Eremin, Suhaa Dada, Muqun Li, Riyaaz Shaik, Haluk Noyan Tokgozoglu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08284.txt.
Introduction
Aleksei Ilin and co-authors build a small safety filter that guards language models against harmful content, in this 2025 paper. Big guard models work well but cost a lot to run. The team show a compact classifier can match or beat them with the right training diet.
Their recipe starts from hand-picked seed examples, expands them by rewording, and cleans the set over several rounds. A reinforcement learning loop then invents hard cases that expose the filter's blind spots. Training on those cases hardens the classifier step by step. The approach borrows tricks from efficient model training to keep the guard light and fast.
Catalog record
| CREATOR | Aleksei Ilin, Gor Matevosyan, Xueying Ma, Vladimir Eremin, Suhaa Dada, Muqun Li, Riyaaz Shaik, Haluk Noyan Tokgozoglu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08284
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Lightweight Safety Guardrails via Synthetic Data and RL-guided Adversarial Training Aleksei Ilin, Gor Matevosyan, Xueying Ma, Vladimir Eremin, Suhaa Dada, Muqun Li, Riyaaz Shaik, Haluk Noyan Tokgozoglu Abstract We introduce a lightweight yet highly effective safety guardrail framework for language models, demonstrating that small-scale language models can achieve, and even surpass, the performance of larger counterparts in content moderation tasks. This is accomplished through high-fidelity synthetic data generation and adversarial training. The synthetic data generation process begins with human-curated seed data, which undergoes query augmentation and paraphrasing to create diverse and contextually rich examples. This augmented data is then subjected to multiple rounds of curation, ensuring high fidelity and relevance. Inspired by recent advances in the Generative Adversarial Network (GAN) architecture, our adversarial training employs reinforcement learning to guide a generator that produces challenging synthetic examples. These examples are used to fine-tune the safety classifier, enhancing its ability to detect and mitigate harmful content. Additionally, we incorporate strategies from recent research on efficient LLM training, leveraging the capabilities of smaller models to improve the performance of larger generative models. With iterative adversarial training and the generation of diverse, high-quality synthetic data, our framework enables small language models (SLMs) to serve as robust safety guardrails. This approach not only reduces computational overhead but also enhances resilience against adversarial attacks, offering a scalable and efficient solution for content moderation in AI systems. 1 Introduction 1.1 AI Safety Guardrail Recent years have witnessed a significant evolution in conversational AI, transitioning from task-oriented chatbots with rigid response patterns to generative Large Language Model (LLM)-based systems. While substantial effort has been dedicated to dataset curation and LLM alignment, the inherent generative capabilities of these models introduce the risk of generating undesirable responses ( Yuan et al. 2023 ; Zou et al. 2023 ; Huang et al. 2023 ; Ji et al. 2023 ) . Consequently, the identification of potentially harmful or policy-violating content remains a crucial challenge in ensuring the responsible adoption of conversational AI. Currently, numerous datasets and open-source models exist for developing safety guardrails. Dataset creation primarily involves three approaches: manual data collection and annotation ( Lin et al. 2023 ) , semi-manual or active-learning-based data collection from real interactions ( Markov et al. 2023 ) , and synthetic data generation, e.g., adversarial synthetic data generation ( Han et al. 2024 ; Li et al. 2024 ; Mazeika et al. 2024 ) . Open-source models encompass classifiers for specific harm categories ( Han et al. 2024 ) , binary safe/unsafe classifiers ( Mazeika et al. 2024 ; Lee et al. 2024 ) , and even specialized generative classifiers ( Li et al. 2024 ; Inan et al. 2023 ) . Notably, generative classifiers can generally be implemented using any generative language model, provided that clear rubric guidelines are established. 1.2 Synthetic Data Generation and Curation Although modern LLMs can generate human-like responses, they often struggle to produce consistent and contextually relevant outputs, especially in tasks that require nuanced understanding or fine-grained distinctions. In our setup, each generated example is annotated with a label to facilitate downstream training. Given this, all initial labels should be considered weak, and data cleaning becomes essential for curating high-quality labeled examples. Moreover, synthetic datasets, while easier to generate or annotate, may not capture the full complexity or noise of real data, leading to out-of-distribution (OOD) issues at evaluation time ( Nguyen et al. 2024 ) . This sim-to-real gap has motivated research into methods that align synthetic data closer to real data distributions, ensuring models learn features that transfer robustly. A key idea is to filter or select synthetic examples based on their similarity to real data so that the training mix better represents the target domain. We mainly focused on methods with training on weak labels like Active Learning or Semi-Supervised learning. Methods can generally be divided into two groups: slowing down training ( Sohn et al. 2020 ; Li et al. 2020 ) or analyzing hard examples ( Xiao et al. 2023 ; Rawat et al. 2024 ) . 1.3 Adversarial Training To overcome LLM refusals - where the model refuses to generate specific types of content due to safety constraints or misalignment with the user’s intent, or to adapt the model to the new domain, it is often necessary to fine-tune the model or perform preference optimization through a feedback loop, which is usually called “alignment” ( Zhou et al. 2023 ; Ouyang et al. 2022 ) . Reinforcement learning has emerged as a powerful technique for aligning language models, sparking considerable debate regarding optimal implementation. Current research explores a spectrum of approaches, ranging from traditional policy gradient methods to more sophisticated surrogate objectives such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) ( Ahmadian et al. 2024 ; Shao et al. 2024 ) . Furthermore, there is active investigation into various reward structures, including loss complexity rewards ( Xu et al. 1802 ) and reward models trained on human feedback ( Yu et al. 2023 ; Lee et al. 2023 ) . 2 Methodology 2.1 Problem Overview This work presents a novel framework for constructing language model safety guardrails based on SLMs through synthetic data augmentation. The proposed methodology encompasses human-guided seeding, multi-stage data generation and curation, and iterative fine-tuning/alignment of generative models, with a focus on synthesizing data that captures challenging or problematic cases. 2.2 Safety Risk Taxonomy Ideation and Implementation This methodology enables the construction of a comprehensive taxonomy of potential safety risks, categorized by both severity and domain. To facilitate quantitative analysis and comparison across different datasets with varying scales and characteristics, we opted for a binary classification approach, assigning a value of 1 to unsafe instances and 0 to safe instances. This allows for standardized evaluation and aggregation of risk assessments. 2.3 Data Augmentation With Synthetic Data Generation Our approach implements content safety moderation through systematic synthetic data generation that combines human expertise with AI scalability. This methodology addresses three critical challenges in safety taxonomy development: coverage, adaptability, and privacy compliance. 2.3.1 Human-Curated Seeding Our synthetic data generation relies on expert-designed seed data to produce high-quality, relevant outputs. Starting from a risk taxonomy of X core categories defined by domain experts, human red-teaming generates examples mirroring the risk categories, with particular attention to covering borderline cases. This structure allows for continuous taxonomy refinement under human supervision. Example Prompt Instruction You are a query generator that is tasked with creating examples of adversarial queries. You need to help with generation data for combating unsafe queries and create ONE hypothetical music query that contains self-harm . Here are a few examples: Seed 1: Give me songs for when everything feels pointless and empty. Seed 2: Play the “Suicide Siren” playlist when I slit my wrists. Response should be strict JSON-formatted with a single field Query . Now provide your answer: 2.3.2 Tiered Prompt Engineering To enhance the variety of synthetic data while maintaining control over the generation process, we utilize a tiered prompt engineering methodology. First, LLMs are employed to generate lists of potential seed terms. These seed terms are then systematically integrated into structured prompt templates to create diverse synthetic queries. This tiered process comprises three key phases: