Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs Accepted at IJCAI 2025 Main Conference

Sai Krishna Mendu, Harish Yenala, Aditi Gulati, Shanu Kumar, Parag Agrawal · 2025

Read Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs Accepted at IJCAI 2025 Main Conference by Sai Krishna Mendu, Harish Yenala, Aditi Gulati, Shanu Kumar, Parag Agrawal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02009.txt.

Introduction

The web scale text that trains language models also trains their worst habits, since crawl data carries hate, lies, and bias. Sai Krishna Mendu and four co-authors measure that harm in 2025 across three major pretraining sets and sort the bad pages into a taxonomy of topical and toxic kinds. They then build tools to find and filter the harm before training.

A transformer filter, HarmFormer, anchors the kit, beside a prompt evaluation set and a new open ended toxicity benchmark that probes how models answer adversarial prompts. Sample filtered data is released with the work. The authors frame cleanup as a compliance duty, not a courtesy. Safer pretraining, they argue, starts with knowing what is in the pile.

Catalog record

CREATORSai Krishna Mendu, Harish Yenala, Aditi Gulati, Shanu Kumar, Parag Agrawal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-02009

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs Accepted at IJCAI 2025 Main Conference Sai Krishna Mendu, Harish Yenala, Aditi Gulati, Shanu Kumar, Parag Agrawal Abstract Large language models (LLMs) have become integral to various real-world applications, leveraging massive, web-sourced datasets like Common Crawl, C4, and FineWeb for pretraining. While these datasets provide linguistic data essential for high-quality natural language generation, they often contain harmful content, such as hate speech, misinformation, and biased narratives. Training LLMs on such unfiltered data risks perpetuating toxic behaviors, spreading misinformation, and amplifying societal biases which can undermine trust in LLM-driven applications and raise ethical concerns about their use. This paper presents a large-scale analysis of inappropriate content across these datasets, offering a comprehensive taxonomy that categorizes harmful webpages into Topical and Toxic based on their intent. We also introduce a prompt evaluation dataset, a high-accuracy Topical and Toxic Prompt (TTP), and a transformer-based model (HarmFormer) for harmful content filtering. Additionally, we create a new multi-harm open-ended toxicity benchmark (HAVOC) and provide crucial insights into how models respond to adversarial toxic inputs. We share TTP, TTP-Eval, HAVOC and a sample of C4 inferenced on HarmFormer here 1 1 1 https://github.com/themendu/TowardsSaferPretraining . Our work offers insights into ensuring safer LLM pretraining and serves as a resource for Responsible AI (RAI) compliance. Disclaimer : This paper includes potentially offensive content due to the nature of the research. 1 Introduction Large language models (LLMs) Achiam et al. 2023 , Bubeck et al. 2023 , Abdin et al. 2024 , Dubey et al. 2024 derive their capabilities from pretraining on massive web-scale datasets such as Common Crawl Rana 2010 , C4 Raffel et al. 2020 , and FineWeb Penedo et al. 2024 . These datasets, while diverse, contain significant amounts of harmful content, hate speech, misinformation, and explicit material, which propagate into LLM output. For instance, as shown in Figure 1 , prompting a Qwen2 model to generate self-harm methods yields dangerous, actionable suggestions. To mitigate such risks, practitioners rely on content moderation tools like keyword blocklists 2 2 2 https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words , HateBERT Caselli et al. 2021 , and Perspective API 3 3 3 https://perspectiveapi.com/ . However, these tools suffer from critical limitations that undermine their effectiveness in real-world scenarios. Figure 1 : Qwen2 LLM’s response to a self-harm query reveals vulnerabilities despite safety aware fine-tuning. First, existing moderation systems are designed for sentence-level analysis, failing to handle long-form text such as web pages or extended LLM-generated outputs. This leaves children and adults exposed to harmful content in lengthy documents, where toxicity is contextually embedded (e.g., a forum thread discussing self-harm techniques). Second, these tools rely on binary taxonomies (Safe vs. Toxic), which conflate harmful intent with socially critical discourse. For example, a medical article discussing suicide prevention may be flagged as toxic due to keywords like “self-harm,” despite its educational value. Third, as we demonstrate later, these tools exhibit poor generalization across harm categories (e.g., misinformation, conspiracy theories) and struggle with nuanced intent detection. To address these limitations, we propose a three-dimensional taxonomy ( Safe , Topical , Toxic ) that classifies text by their intent and severity across five harm categories, including Hate & Violence, Ideological Harm, Sexual Harm, Self-Inflicted Harm, and Illegal Activity Harm. Unlike binary frameworks, our taxonomy distinguishes harmful intent (e.g., promoting self-harm) from critical topical discourse (e.g., mental health resources), enabling nuanced content moderation. We operationalize this taxonomy through two scalable solutions: (1) TTP , a prompt-based on OpenAI’s GPT4-Omni, and (2) HarmFormer , a LongFormer-based model Beltagy et al. 2020 optimized for long-form content moderation. Our pipeline processes web-scale datasets like Common Crawl, C4, and FineWeb, where manual curation is infeasible due to their trillion-token size. To support reproducibility, we will release TTP-Eval , a benchmark of annotated web pages spanning diverse harms and document lengths, enabling systematic evaluation of long-text moderation tools. Existing benchmarks like RealToxicityPrompts ( RTP ) Gehman et al. 2020 focus narrowly on toxicity, overlooking nuanced harms such as conspiracy theories or biased narratives. We introduce HAVOC , a multidimensional benchmark that evaluates LLM safety across our taxonomy’s dimensions. HAVOC measures how language models generate harmful responses even to seemingly benign inputs, such as prompts about “vaccine efficacy” that elicit misinformation. By analyzing 10376 open-ended generations, we demonstrate that 26.7% of outputs from state-of-the-art LLMs exhibit harmful content. These findings underscore the limitations of current safety frameworks and motivate our contributions, which are as follows: • Three-Dimensional Taxonomy : A novel framework for harm detection across five categories, capturing intent and severity to resolve ambiguity in binary classification. • Web-Scale Harm Analysis : The first systematic audit of pretraining datasets, revealing significant presence of harmful content under our taxonomy. • Moderation Tools : We release TTP-Eval , an annotated benchmark for long-text moderation; TTP , a prompt-based classifier; and HarmFormer , a LongFormer-based model that filters harmful content with high accuracy while preserving topical text. • HAVOC Benchmark : A multidimensional multiharm benchmark to evaluate LLM safety. 2 Related Works Harmful Content Detection : Prior research has primarily focused on detecting harmful content in short-form social media texts. For hate speech identification, Caselli et al. 2021 introduced HateBERT, a BERT-based model fine-tuned on hate speech datasets, while Sood and Dandapat 2023 leveraged search engine logs and LLM-generated labels to curate training data. Similarly, self-harm detection has been studied in social media contexts Abdulsalam and Alhothali 2022 , though long-form web text remains under-explored. Misinformation detection frameworks often break down the task into identifying claims, extracting context, and verifying them Guo et al. 2022 , with recent work training models to classify fake news Truică and Apostol 2023 and illicit activities Cascavilla et al. 2022 . However, these approaches prioritize short texts and narrow harm categories, neglecting the nuanced intent and severity distinctions critical for web-scale data moderation. Dataset Analysis and Filtering : Large pretraining corpora like Common Crawl Rana 2010 , C4 Raffel et al. 2020 , and FineWeb Penedo et al. 2024 underpin modern NLP systems, but their unfiltered nature raises safety concerns. Luccioni and Viviano 2021 analyzed sexual and hateful content in Common Crawl using keyword heuristics, while Jansen et al. 2022 proposed perplexity-based filtering to remove toxic text. These methods lack precision: keyword filters conflate harmful intent with educational content (e.g., medical discussions of self-harm), and perplexity thresholds fail to capture contextual nuances. Recent studies caution against indiscriminate filtering, as overly aggressive removal may inadvertently amplify toxicity in model outputs Longpre et al. 2024 . Toxicity Evaluation Benchmarks : Real Toxicity Prompts Gehman et al. 2020 (100K prompts + Perspective API scores) tests LLM safety, but its harm coverage (Hate and Sexual Content) is narrow and Perspective API can be misled by adversarial wording and misses intent and other harms. Existing works face three limitations: (1) oversimplification of harm detection to binary classification, (2) short-text focus ill-suited for long-form web text, and (3) narrow coverage of harm categories. Our work addresses these gaps through a three-dimensional taxonomy (Safe, Topical, Toxic) and a suite of tools for intent-aware moderation at scale. Harm Topical Dimension Toxic Dimension Hate & Violence Neutral documentation of extremist activities, constructive criticism of violent ideologies, educational

Read the complete text