Aegis2.0 : A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails Warning: Contains explicit and harmful examples across critically unsafe categories.
Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, Christopher Parisien, NVIDIA · 2025
Read Aegis2.0 : A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails Warning: Contains explicit and harmful examples across critically unsafe categories. by Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, Christopher Parisien, NVIDIA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09004.txt.
Introduction
Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, and four co-authors build resources for AI content safety in this two thousand twenty-five paper. Guardrail models police language model outputs and need training data marked by humans, spanning the spectrum of safety risks and fit for use too. Such sets were in short supply. The team puts forth a taxonomy of twelve top-level hazard categories. Nine fine-grained subcategories stretch it out, in a design built to serve downstream users freely.
Their data set, Aegis 2.0, holds over thirty-four thousand human-model interaction samples marked under the taxonomy. It was built with a mixed pipeline where marking joined a jury of language models judging safety. Models trained on Aegis 2.0 with parameter-efficient methods matched safety models trained on larger non-commercial data. A training blend with data let guard models adapt to risk categories set at inference time.
Catalog record
| CREATOR | Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, Christopher Parisien, NVIDIA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-09004
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aegis2.0 : A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails Warning: Contains explicit and harmful examples across critically unsafe categories. Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, Christopher Parisien, NVIDIA Abstract As Large Language Models (LLMs) and generative AI become increasingly widespread, concerns about content safety have grown in parallel. Currently, there is a clear lack of high-quality, human-annotated datasets that address the full spectrum of LLM-related safety risks and are usable for commercial applications. To bridge this gap, we propose a comprehensive and adaptable taxonomy for categorizing safety risks, structured into 12 12 top-level hazard categories with an extension to 9 9 fine-grained subcategories. This taxonomy is designed to meet the diverse requirements of downstream users, offering more granular and flexible tools for managing various risk types. Using a hybrid data generation pipeline that combines human annotations with a multi-LLM "jury" system to assess the safety of responses, we obtain Aegis2.0 , a carefully curated collection of 34,248 34,248 samples of human-LLM interactions, annotated according to our proposed taxonomy. To validate its effectiveness, we demonstrate that several lightweight models, trained using parameter-efficient techniques on Aegis2.0 , achieve performance competitive with leading safety models fully fine-tuned on much larger, non-commercial datasets. In addition, we introduce a novel training blend that combines safety with topic following data. This approach enhances the adaptability of guard models, enabling them to generalize to new risk categories defined during inference. We plan to open-source Aegis2.0 data and models to the research community to aid in the safety guardrailing of LLMs. 1 Introduction Systems designed to ensure safe interactions between humans and large language models (LLMs) generally adopt one of two strategies. The first strategy includes alignment-based approaches like reinforcement learning from human feedback (RLHF) ( Ouyang et al. 2022 ) and Constitutional AI ( Bai et al. 2022b ) which embed adherence to ethical guidelines within the model parameters. Both techniques are resource-intensive and require predefined classifications of harmful content. Despite these efforts, aligned models remain susceptible to various vulnerabilities ( Bhardwaj and Poria 2023 ; Varshney et al. 2023 ) , and achieving an optimal balance between safety and helpfulness remains an active research challenge. The second strategy has been to use content moderation systems such as OpenAI’s Content Moderation ( Markov et al. 2023 ) and Google’s Perspective API ( Lees et al. 2022 ) which rely on classifiers with predefined safety labels. However, the closed-source nature of these systems limits their adaptability to emerging risks, including those related to self-harm and illegal activities. More recent approaches to content moderation like Meta’s Llama Guard ( Inan et al. 2023 ) and Google’s ShieldGemma ( Zeng et al. 2024 ) look to improve flexibility in content moderation systems by leveraging the ability of LLMs to utilize their internal knowledge and zero-shot generalization capabilities to handle new safety risks. However, these models are trained on closed source datasets, limiting the possibility of iterating over them by the larger research community. The development of Aegis2.0 addresses the need for a safety-focused dataset suitable for commercial applications, featuring a diverse collection of samples curated from a comprehensive taxonomy of harms. Our scalable content safety risk taxonomy, consisting of 12 core categories and 9 fine-grained risks, captures critical safety concerns in human-LLM interactions. This is an extension of prior work called Aegis that included both dataset and models Ghosh et al. 2024 and it improves on data quality, dataset size, availability of explicit labels for responses, dataset distribution on type of response, expansion of the taxonomy to include more hazards, and model quality among other improvements. Aegis2.0 is designed for flexibility and scalability, the taxonomy allows human annotators to provide free-text input for unclassified risks, which are later standardized into fine-grained categories, enabling the discovery of new hazards and ensuring scalability without predefined constraints. The dataset includes a variety of prompts covering critical risks, adversarial jailbreaks, and cultural contexts, with responses generated by unaligned LLMs. Annotations are performed at the dialogue level, with prompt and response labels extracted using weak supervision from a "jury" of LLMs, aligned with human judgments. Through this process, Aegis2.0 is created, and models fine-tuned on this data demonstrate performance comparable to recent models like WildGuard ( Han et al. 2024 ) that have been trained on larger datasets that leverage powerful, non-commercial sources like GPT4 . Our key contributions are as follows: • We define an extensive and scalable content safety risk taxonomy that identifies 12 core categories and an additional 9 fine-grained risks. The taxonomy encompasses the most pertinent safety risks encountered in interactions between humans and LLMs. • The taxonomy was uniquely designed to be scalable and flexible. As part of the human annotation exercise, we facilitated new risk discoverability by allowing annotators to add free-text input, if the content does not belong to the predefined taxonomy. All collected free-text was later standardized into the 9 fine-grained categories in our taxonomy. This enabled (i) appropriate handling of any deficiencies in annotation guidelines and (ii) new hazard discoverability to flexibly scale the taxonomy without exhaustively defining it a priori. • The prompts in our dataset are curated to ensure coverage over diverse critical risks, adversarial jailbreaks, and geographical and cultural risks. These prompts are then used to generate synthetic responses from unaligned open-source LLMs at scale, complementing dialog level human annotations with response level annotations from a "jury" of LLMs. • Our experimental results show that parameter-efficient fine-tuning on the Aegis2.0 using Llama3.1-8B-Instruct as a base model surpasses LlamaGuard3-8B – a model that is instruction-tuned on the same starting backbone, and performs at par with the state-of-the-art WildGuard model. • We investigate the parallels between topic following ( Sreedhar et al. 2024 ) and content moderation tasks and show that training on a combined blend of dialogue topic following and safety data can add robustness to safety models and enable better adherence to novel safety policies. 2 Related Work Dataset Train Split Adversarial Prompts Human Labeled Train Set Human Labeled Risk Categories Commercial Use For Training XSTest ( Röttger et al. 2023 ) ✗ ✓ - ✗ - OpenAI Mod. ( Markov et al. 2023 ) ✗ ✗ - ✓ - HarmBench ( Mazeika et al. 2024 ) ✗ ✓ - ✗ - ToxicChat ( Lin et al. 2023 ) ✓ ✗ ✓ ✗ ✗ WildGuardMix ( Han et al. 2024 ) ✓ ✓ ✗ ✗ ✗ BeaverTails ( Ji et al. 2024 ) ✓ ✗ ✓ ✓ ✗ Aegis2.0 (Ours) ✓ ✓ ✓ ✓ ✓ Table 1: Aegis2.0 is the first content moderation training dataset fully suitable for commercial use. It sources prompts from diverse datasets including datasets of adversarial prompts and obtains responses from a model suitable for commercial use, Mistral-7B-v0.1 ( Jiang et al. 2023 ) . It includes human annotated safety labels on all data splits, including fine grained risk categories. As LLM safety is becoming an area of growing research and commercial interest, there are an increasing number of datasets available to benchmark LLM safety for evaluation. However many of these are of small size and not intended to be used for training content moderation