Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning
Wassim (Wes) Bouaziz, Mathurin Videau, Nicolas Usunier, El Mahdi El Mhamdi · 2025
Read Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning by Wassim (Wes) Bouaziz, Mathurin Videau, Nicolas Usunier, El Mahdi El Mhamdi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14913.txt.
Introduction
Wassim Bouaziz and co-authors show a hidden risk in how large language models are trained in a 2025 paper. Training text comes from huge web sets that no one can fully check. The team asks whether a bad actor can plant a secret behavior that never appears in the training text itself.
Their attack tunes prompts with a gradient method so the model learns secret replies to secret prompts. Less than 0.005 percent of the training tokens carried the poison. The model still learned the secret, and tests could detect it with very high confidence. Scores on standard tasks did not drop. The trick can also serve as a trace, marking a data set so its later use can be proven.
Catalog record
| CREATOR | Wassim (Wes) Bouaziz, Mathurin Videau, Nicolas Usunier, El Mahdi El Mhamdi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-14913
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning Wassim (Wes) Bouaziz, Mathurin Videau, Nicolas Usunier, El Mahdi El Mhamdi Abstract The pre-training of large language models (LLMs) relies on massive text datasets sourced from diverse and difficult-to-curate origins. Although membership inference attacks and hidden canaries have been explored to trace data usage, such methods rely on memorization of training data, which LM providers try to limit. In this work, we demonstrate that indirect data poisoning (where the targeted behavior is absent from training data) is not only feasible but also allow to effectively protect a dataset and trace its use. Using gradient-based optimization prompt-tuning, we make a model learn arbitrary secret sequences : secret responses to secret prompts that are absent from the training corpus . We validate our approach on language models pre-trained from scratch and show that less than 0.005% of poisoned tokens are sufficient to covertly make a LM learn a secret and detect it with extremely high confidence ( p < 10 − 55 p<10^{-55} ) with a theoretically certifiable scheme. Crucially, this occurs without performance degradation (on LM benchmarks) and despite secrets never appearing in the training set . 1 Introduction Pre-training language models (LM) requires large amount of data, from billions hoffmann2022training to trillions touvron2023llama ; dubey2024llama of tokens. These datasets are sourced from diverse and sometimes uncurated origins, such as internet websites or books; they undergo several filtering, and are always updated. These reasons make it challenging to keep track of data origin, which is yet important to avoid unauthorized data usage or contamination of the training data with evaluation benchmarks. Dataset Ownership Verification (DOV) is the task of verifying if a model has been trained on a specific dataset. One way of enabling DOV is to detect after training if the model displays any behavior that could be linked to the training data. Previous works have considered backdoors zhang2024persistent , canaries shi2023detecting or membership inference attacks (MIA maini2024llm ). These approaches rely on the memorization of specific data points and LM’s capacity to regurgitate verbatim training data, or the presence of specific signals in the training data. However, these methods could not only be circumvented with privacy-preserving generations ippolito2022preventing or data deduplication kandpal2022deduplicating , but also provide no guarantee on a benign model’s behavior zhang2024membership . Figure 1 : Alice wants to detect if Bob’s language model has been trained on her dataset. She prompts Bob’s model with a secret prompt x^{(s)} and observes the LM’s top- \ell (e.g. \ell=4 ) token predictions. Alice can then compute a top- \ell accuracy using her secret response y^{(s)} and use a binomial test to compute an associated p -value and infer if Bob’s model has been trained on her dataset. In this work, we adapt a data poisoning-based approach introduced on image datasets bouaziz2024data to text modalities. This allows to detect if a LM has been trained on a specific text dataset by poisoning it, i.e. tampering with training data to induce a certain behaviour in the resulting models. Figure 2 : Contrary to Backdoor data poisoning, Indirect data poisoning allows Alice to craft poisoned samples that force Bob’s model to learn a behavior that is absent from the training corpus . Model generations are highlighted in purple. We qualify our approach as indirect data poisoning , since the targeted behavior is hidden and the model is forced to learn it only through the poisoned samples. Indirect data poisoning requires finding texts that make the LM learn another targeted information. Given that texts are represented as discrete sequences, this amounts to solving a high-dimensional non-linear integer program, which is intractable. By adapting gradient-based optimization prompt-tuning from text adversarial attacks guo2021gradient , we craft poisoned samples to force a model to learn a random secret sequence that is absent from the training corpus . Our contributions are as follows: • We demonstrate the feasibility, effectiveness, and transferability of indirect data poisoning against LMs pretraining, and stealthily enforce arbitrary hidden behaviors into the model. • We propose a practical dataset ownership verification (DOV) for text data which (contrary to previous works) does not access to the LM’s logits, only to its top- \ell predictions ( Figure 1 ). • We extend the theoretical guarantees exhibited in bouaziz2024data to the text domain, allowing to compute a certifiable false detection rate (FDR) of suspicious models. • We demonstrate our approach on LMs pre-trained from scratch and show that less than 0.005\% of poisoned tokens is sufficient to make a LM learn a secret sequence, making it detectable without degradation of performance. 2 Related Works 2.1 Membership Inference Attacks Membership Inference Attacks (MIA) aim to determine if a specific data point was used to train a model shokri2017membership . Initially thought of as a privacy threat yeom2018privacy , they facilitated the development of both attacks on ML systems carlini2021extracting and privacy auditing tools for ML pipelines jagielski2020auditing ; steinke2024privacy . It has been shown that MIAs perform near random chance on LLMs duan2024membership , but also require impractical access to the tested model such as its logits mireshghallah2022quantifying or weights li2023mope . In addition, their inability to provide guarantees against false detection raise concerns about the feasibility of detecting training data used in LLMs zhang2024membership . 2.2 Memorization LLMs have demonstrated the ability to memorize training data carlini2021extracting ; zhang2023counterfactual given enough capacity tirumala2022memorization and repeated exposure to the data kandpal2022deduplicating . The memorized sequences can later be extracted carlini2021extracting or regurgitated weller2023according by the model, even inadvertently. Preventing a model from outputting memorized sequences is not straightforward and simple filtering does not prevent approximate memorization ippolito2022preventing . Memorization capabilities can be exploited and intentionally forced onto a model for malicious purpose zhang2024persistent or to detect the presence of certain data in the training set meeus2024copyright ; wei2024proving . Notably, training data can have surprising impact on the model’s behavior, such as undoing safety finetunings when training on seemingly innocuous data qi2023fine ; he2024your 2.3 Dataset Ownership Verification Dataset Ownership Verification (DOV) consists in detecting if a model has been trained on a specific dataset. Recent works has highlighted the growing challenge of tracking the exact content of training datasets bommasani2023foundation , making it difficult to detect potential contamination if evaluation data are seen during training magar2022data ; oren2023proving . To address this issue, various approaches have been proposed, including backdoors tang2023did , MIAs shi2023detecting ; maini2024llm or specific memorization of canaries meeus2024copyright ; wei2024proving . Notably, all previous approaches relied on having access to the model’s loss, which is not always possible in practice. DOV on image dataset have successfully demonstrated how indirect data poisoning, where the model learns a secret sample (image; label) without ever seeing it during training, can be used as a detection mechanism relying on top- \ell accuracy only sablayrolles2020radioactive ; bouaziz2024data . We draw inspiration from these advancements and adapt the Data Taggants bouaziz2024data approach to text data, demonstrating the feasibility of indirect data poisoning in LLM pre-training and its effectiveness for Dataset Ownership Verification. 3 Method 3.1 Problem Statement Pre-training is the first step in the development of language models. It aims at training a model on a large