Safety Pretraining: Toward the Next Generation of Safe AI

Pratyush Maini Sachin Goyal Dylan Sam, Alex Robey Yash Savani Yiding Jiang Andy Zou, Matt Fredrikson Zachary C. Lipton J. Zico Kolter · 2025

Read Safety Pretraining: Toward the Next Generation of Safe AI by Pratyush Maini Sachin Goyal Dylan Sam, Alex Robey Yash Savani Yiding Jiang Andy Zou, Matt Fredrikson Zachary C. Lipton J. Zico Kolter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16980.txt.

Introduction

Safety tuning after training can fail, since a model may learn unsafe habits deep in its first pass over the web. Pratyush Maini and co-authors build safety into that first pass in a 2025 paper. Their plan sorts web text by safety, rewrites unsafe text into safer tales, and adds refusal lessons.

A marked token also flags unsafe text during training, so the model can be steered at use time. Attack success on standard safety tests fell from 38.8 percent to 8.4 percent. Scores on general tasks did not drop. The team frames the work as a root-level fix, not a patch. Data, not just rules, carries the safety lesson.

Catalog record

CREATORPratyush Maini Sachin Goyal Dylan Sam, Alex Robey Yash Savani Yiding Jiang Andy Zou, Matt Fredrikson Zachary C. Lipton J. Zico Kolter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16980

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Safety Pretraining: Toward the Next Generation of Safe AI Pratyush Maini Sachin Goyal Dylan Sam, Alex Robey Yash Savani Yiding Jiang Andy Zou, Matt Fredrikson Zachary C. Lipton J. Zico Kolter Abstract As large language models (LLMs) are increasingly deployed in high-stakes settings, the risk of generating harmful or toxic content remains a central challenge. Post-hoc alignment methods are brittle: once unsafe patterns are learned during pretraining, they are hard to remove. In this work, we present a data-centric pretraining framework that builds safety into the model from the start. Our framework consists of four key steps: (i) Safety Filtering: building a safety classifier to classify webdata into safe and unsafe categories; (ii) Safety Rephrasing: we recontextualize unsafe webdata into safer narratives; (iii) Native Refusal: we develop RefuseWeb and Moral Education pretraining datasets that actively teach model to refuse on unsafe content and the moral reasoning behind it, and (iv) Harmfulness-Tag annotated pretraining: we flag unsafe content during pretraining using a special token, and use it to steer model away from unsafe generations at inference. Our safety-pretrained models reduce attack success rates from 38.8% to 8.4% on standard LLM safety benchmarks with no performance degradation on general tasks. 1 Introduction Artificial intelligence (AI) increasingly permeates critical sectors such as healthcare, education, and public policy. While this broad adoption highlights AI’s potential, it also amplifies risks related to generating and propagating harmful or toxic content. Traditional post-hoc alignment techniques such as Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ) , Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) , and Constitutional AI ( Bai et al., 2022b ) have been proposed to align AI models. However, adversarial analyses of models trained using these techniques reveal that the reduction in toxic content generation is, at best, superficial ( Zou et al., 2023 ; Chao et al., 2023 ) . Once a model internalizes unsafe information, the vast literature surrounding LLM unlearning indicates that it is difficult, if not impossible, to unlearn it ( Zhou et al., 2023 ; Maini et al., 2024a ; Li et al., 2024 ; Schwarzschild et al., 2024 ) . In short, alignment is not unlearning. This observation underscores our approach: rather than relying solely on post-hoc alignment methods, we embed safety directly into the pretraining process through a comprehensive, data-centric strategy. The key contributions that led us to the development of the natively safe family of SafeLM models were: (i) A renewed focus on safety filtering, along with the establishment of a practice for Data Safety Report Cards; (ii) Synthetic re-contextualization to ensure potentially harmful content is taught in a safe and context-rich environment; (iii) Native refusal training with moral education books, and refusal conversations to teach the model to steer away from bad actors; (iv) Harmfulness-Tag tagging for unsafe content to allow LLMs to better separate safe and unsafe content; (v) Evaluation benchmarks that enable safety testing of base models that have not yet been safety tuned. Safety Filtering We introduce a robust safety filtering mechanism that proactively excises harmful content from training data. Central to this effort is our Safety Classifier , which was trained on 10K samples annotated by GPT-4 using a highly specific prompting protocol (detailed in Section 3.1 ). Unlike legacy models—such as the roberta_toxicity_classifier developed in 2019, and being used in modern LLM pretraining corpora ( Soldaini et al., 2024 ) —our lightweight classifier is designed for broad adoption, with the potential to serve as a de-facto standard for subsequent pre-training corpus curation. Accompanying this classifier is a suite of Annotated Data that refines our filtering pipeline and sets a new benchmark for safe pre-training practices. In accordance with this new focus on ensuring safe pretraining data, we establish a new practice of providing Data Safety Report Cards alongside any dataset release, which captures the safety score of that dataset alongside frequencies of harmful text from various categories from the MLCommons safety taxonomy ( Vidgen et al., 2024 ) . Synthetic Recontextualization Removing unsafe content entirely risks discarding valuable information. To address this, we employ a synthetic recontextualization strategy, wherein potentially harmful data is not simply excised but carefully rephrased and reframed. This method preserves the underlying information while embedding it in a context that underscores its ethical and historical implications (see Section 3.2 ). Notably, we release SafeWeb , a safety-focused 100 billion token synthetic data corpus constructed using 12 diverse prompts (ranging from podcasts to classroom lectures). This diversity ensures that, for instance, the model learns about critical historical events like the Holocaust while fully appreciating their tragic consequences. Native Refusal Training We introduce RefuseWeb , a dataset that simulates refusal scenarios based on real harmful data (scores 4 and 5), and guides models to responsibly disengage from harmful prompts (Section 3.3 ). To generalize these ethical patterns beyond dialogue, we also introduce Moral Education datasets that reframe harmful content into cohesive educational formats (Section 3.4 ). Harmfulness-Tag Annotated Training and Inference To further enhance the separation of safe and unsafe representations, we incorporate harmfulness-tag annotated training. In our framework, a harmfulness-tag (e.g., <potentially_unsafe_content> ) is inserted into the training data to signal potentially harmful passages. This annotation strategy informs our SafeBeam inference algorithm, which dynamically selects the next token in a way that minimizes the probability of the harmfulness-tag being generated (as detailed in Section 4 ). New Evaluation Tools Finally, we present new evaluation tools that enable safety testing of base model behaviors that previous evaluation frameworks could not provide ( Ouyang et al., 2022 ) . These tools allow us to measure safety under ‘completions’ as opposed to chat completions, enabling the monitoring of safety during pre-training. Our work culminates in the release of the SafeLM , a 1.7B parameter natively safe LLM, alongside an interactive Safe Playground hosted on Hugging Face. These openly accessible resources aim to democratize and accelerate AI safety research. Empirically, our Safety-Native Pretraining significantly reduces the rate of harmful generations, achieving a reduction in attack success rate (ASR) from 38.8% to 8.3% on safety benchmarks. Crucially, this enhancement does not come at the expense of performance on standard NLP benchmarks, though we do observe modest impacts on helpfulness via over-refusal as discussed in Section 6 . In summary, by fundamentally embedding safety during pretraining rather than post-hoc tuning, our framework sets a robust foundation for developing AI systems that are inherently safer, ethically sound, and better aligned with human values. 2 Related Work Post-training alignment. Ensuring the safety of LLMs has become a focal point in AI research. Various strategies have been proposed to limit the generation of harmful, toxic, or otherwise objectionable content. The majority of this effort has involved the design of bespoke post-training algorithms—including RLHF ( Ouyang et al., 2022 ) and supervised fine-tuning on curated datasets ( Bai et al., 2022a ) —which tweak an LLM’s propensity to refuse to respond to harmful queries. However, these approaches tend to be resource intensive, require large amounts of annotated preference data, and often fail to prevent the extraction of harmful content, especially under adversarial pressure ( Zou et al., 2023 ; Chao et al., 2023 ) . More recently, Shi et al. (2023)

Read the complete text