Detecting and Filtering Unsafe Training Data via Data Attribution

Yijun Pan Taiwei Shi Jieyu Zhao Jiaqi W. Ma · 2025

Read Detecting and Filtering Unsafe Training Data via Data Attribution by Yijun Pan Taiwei Shi Jieyu Zhao Jiaqi W. Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11411.txt.

Introduction

Yijun Pan and co-authors tackle a quiet risk in training large language models: a small share of unsafe text in the training set can skew how a model acts. Their 2025 study seeks bad samples by likeness to a small set of known unsafe examples, rather than by training a costly filter model.

The team found that likeness checks get muddied by harmless words inside unsafe texts. Their fix cleans the inner form of both sets before matching, dropping the noise. On tests for jailbreak text and gender bias, the cleaned match beat top filter methods. Model builders gain a cheaper, more flexible way to sweep training data for harm.

Catalog record

CREATORYijun Pan Taiwei Shi Jieyu Zhao Jiaqi W. Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11411

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detecting and Filtering Unsafe Training Data via Data Attribution Yijun Pan Taiwei Shi Jieyu Zhao Jiaqi W. Ma Abstract Large language models (LLMs) are vulnerable to unsafe training data that even small amounts of unsafe data can lead to harmful model behaviors. Detecting and filtering such unsafe training data is essential for trustworthy model development. Current state-of-the-art (SOTA) approaches typically rely on training moderation classifiers which requires significant computational overhead and are limited to predefined taxonomies, making them less adaptable to evolving safety concerns. Moreover, these classifiers lack insight into the training process, limiting their effectiveness in filtering unsafe data. To address these limitations, we propose DABUF, leveraging data attribution to detect and filter unsafe training data by attributing harmful model outputs to influential training data points. DABUF enables flexible identification of various unsafe data types without predefined taxonomies. However, in practice, model outputs can be complex with combined safe linguistic features and unsafe content, leading to reduced attribution accuracy. In such cases, DABUF will integrate moderation classifiers to identify a minimal subset of unsafe training data for targeted attribution (such as jailbreak). When model outputs are relatively straightforward, DABUF uses model outputs directly as the attribution targets. We evaluate the performance on two different tasks: in filtering jailbreaking training data and in identifying and mitigating gender bias. DABUF outperforms SOTA approaches by up to 7.5% in detection AUPRC in jailbreaking scenarios, and 44.1% in detecting gender bias. Moreover, retraining on DABUF-filtered data leads to higher model safety across experiments, underscoring its versatility in addressing a broad spectrum of unsafe data issues. 1 Introduction Large Language Models (LLMs) are known to exhibit various unsafe behaviors, including toxicity, stereotyping, privacy leaks, and ethical violations ( Wang et al. 2024a ) . A primary source of these issues is unsafe training data ( Jiang et al. 2024 ; Chen et al. 2024 ) . For instance, inherent biases or toxic content in a dataset can lead to harmful responses ( Jiang et al. 2024 ; Ouyang et al. 2022 ) , while deliberate attacks, such as adversarial prompts or injected backdoors, can be used to bypass safety alignments ( Chen et al. 2024 ; Zou et al. 2023 ; Li et al. 2024b ) . Consequently, identifying and removing these unsafe training instances is critical for mitigating risks and building safer LLMs. Existing methods for detecting and filtering unsafe training data typically rely on moderation classifiers. Online API tools, such as the Perspective API 1 1 1 https://www.perspectiveapi.com/ and OpenAI’s Moderation API ( Markov et al. 2023 ) , focus on certain predefined toxicity taxonomies, but struggle to generalize to nuanced and emerging unsafe artifacts beyond these predefined taxonomies Weber et al. 2025 . Fine-tuned detection models, including Llama-Guard-3-8B ( Llama Team 2024 ) and Wildguard ( Han et al. 2024 ) , require significant time and resources for additional data collection and training. Moreover, these moderation classifiers primarily detect semantically unsafe training data without considering the influence of each data point on model training, resulting in suboptimal filtering effectiveness for enhancing model safety. In this work, we introduce D ata- A ttribution- B ased U nsafe Training Data Detection and F iltering (DABUF), a method that leverages data attribution techniques to enhance unsafe data detection and filtering in LLMs. Data attribution is a family of methods that quantify the influence of individual training data points on specific model outputs ( Koh and Liang 2020 ; Pruthi et al. 2020 ) . Our central hypothesis is that unsafe training data exerts greater influence on unsafe outputs; thus, attributing these outputs back to their influential training instances can reveal which data points are responsible. These methods do not require additional training data and can be applied flexibly to diverse types of unsafe model outputs. However, applying data attribution naively—by directly attributing unsafe model generations to their training data—is ineffective in many cases. LLM outputs, particularly in long-form generations such as jailbreaking attacks, are influenced by a mix of benign and unsafe training data. Since model generations include common linguistic structures (e.g., stop words, neutral phrases) alongside unsafe content, direct attribution to the entire sequence leads to a noisy attribution signal, reducing the precision of unsafe data identification. To address this challenge, our method introduces a targeted filtering mechanism. Specifically, for long-form outputs, such as those found in jailbreaking scenarios, we first use moderation classifiers to identify a small subset of clearly unsafe training data. We then use this subset as the attribution target, allowing us to refine the attribution process and isolate the most influential unsafe training instances. In contrast, for shorter model outputs, such as those in gender bias scenarios, where the influence of training data is more direct and less noisy, standard data attribution techniques are sufficient without additional moderation filtering. We validate our approach through experiments in two distinct setups. In jailbreaking scenarios involving adversarial prompts that lead to noisy or long unsafe outputs, we apply the proposed DABUF with moderation classifiers for initial unsafe data identification. Conversely, in the gender bias scenario—where outputs are relatively concise—we directly apply DABUF on the model generation. Experimental results show that our approach achieves superior detection performance across different model architectures in jailbreaking scenarios and deliver improved safety when models are retrained with filtered data, outperforming state-of-the-art detection methods. Furthermore, our method generalizes effectively to gender bias scenarios, highlighting its versatility. 2 Related work 2.1 Sources of Unsafe Training Data in LLMs Recent studies ( Yi et al. 2024 ; Qi et al. 2023 ) reveal that malicious fine-tuning can severely compromise safety alignment, even with limited exposure to unsafe data. Unfortunately, current online fine-tuning services are inadequate at detecting these unsafe training data, leaving LLMs vulnerable to potential exploitation ( Qi et al. 2023 ) . Unsafe data may also emerge from synthetic training data generation. For instance, Wang et al. 2022 generate samples by conditioning LLMs on specific keywords and target labels, while Wang et al. 2023 synthesize fine-tuning data from LLM-generated responses. However, as recent safety research ( Wang et al. 2024a ) indicates, even highly aligned models like GPT-4 and GPT-3.5 exhibit unsafe behaviors, suggesting that synthetic data can introduce significant risks. In addition, inherent biases in training data pose challenges that current detection methods are not equipped to handle. Studies have shown that gender bias in training data can lead LLMs to develop skewed assumptions about occupations ( Kotek et al. 2023 ) , while cognitive biases during training undermine model reliability in high-stakes decisions ( Itzhak et al. 2024 ; Echterhoff et al. 2024 ) . The diverse source of unsafe training data highlights the need for more flexible and adaptable detection methods. Current moderation classifiers, often designed for specific content moderation tasks, are insufficient for addressing the complexity and variability of unsafe data in training pipelines. 2.2 Unsafe Training Data Detection in LLMs Existing efforts to detect unsafe training data primarily focus on content moderation classifiers. For example, online moderation tools such as OpenAI’s Moderation API ( Markov et al. 2023 ) are developed to detect harmful content. Recently, there has been growing efforts in developing LLM-based classifiers. One line of research has explored fine-tuning open-source

Read the complete text