Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Ken Ziyu Liu, Christopher A. Choquette-Choo, Matthew Jagielski, Peter Kairouz, Sanmi Koyejo, Percy Liang, Nicolas Papernot · 2025
Read Language Models May Verbatim Complete Text They Were Not Explicitly Trained On by Ken Ziyu Liu, Christopher A. Choquette-Choo, Matthew Jagielski, Peter Kairouz, Sanmi Koyejo, Percy Liang, Nicolas Papernot (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17514.txt.
Introduction
Ken Ziyu Liu and colleagues show that the standard test for whether a text trained a language model can be fooled by construction. The test asks the model to complete a passage; success is taken as membership. Membership itself is usually defined by word-overlap with the training set. Writing in twenty twenty-five, the team retrains models from scratch after deleting every sample the model could complete. Completions still happen. Exact duplicates, near-duplicates, and even short overlaps sneak the information back in. No single overlap threshold separates members from non-members cleanly.
Worse, the team builds adversarial data sets that make a target passage completable without containing it, for any reasonable threshold. Membership, they argue, fails to account for everything the training algorithm can extract. Audits of training data need a richer definition than shared word runs.
Catalog record
| CREATOR | Ken Ziyu Liu, Christopher A. Choquette-Choo, Matthew Jagielski, Peter Kairouz, Sanmi Koyejo, Percy Liang, Nicolas Papernot |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-17514
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language Models May Verbatim Complete Text They Were Not Explicitly Trained On Ken Ziyu Liu, Christopher A. Choquette-Choo, Matthew Jagielski, Peter Kairouz, Sanmi Koyejo, Percy Liang, Nicolas Papernot Abstract An important question today is whether a given text was used to train a large language model (LLM). A completion test is often employed: check if the LLM completes a sufficiently complex text. This, however, requires a ground-truth definition of membership; most commonly, it is defined as a member based on the n n -gram overlap between the target text and any text in the dataset. In this work, we demonstrate that this n n -gram based membership definition can be effectively gamed. We study scenarios where sequences are non-members for a given n n and we find that completion tests still succeed. We find many natural cases of this phenomenon by retraining LLMs from scratch after removing all training samples that were completed; these cases include exact duplicates, near-duplicates, and even short overlaps. They showcase that it is difficult to find a single viable choice of n n for membership definitions. Using these insights, we design adversarial datasets that can cause a given target sequence to be completed without containing it, for any reasonable choice of n n . Our findings highlight the inadequacy of n n -gram membership, suggesting membership definitions fail to account for auxiliary information available to the training algorithm. † † affiliationnotice: Equal contribution 1 Introduction Training data membership asks whether a data point was used to train a given model. For large language models (LLMs), it is useful to answer questions related to privacy (e.g., is the LLM leaking information contained in the text it was trained on?), copyright (e.g., has the model been trained on copyrighted text?), and more generally AI safety (e.g., did the LLM successfully unlearn text that was identified as harmful post hoc training?). In each of these settings, it is important that the evaluation of membership be robust. With direct access to the training dataset, determining sequence membership is straightforward given a fixed criterion for defining when two text sequences are “the same.” A common approach is to compare sequences by their n -grams, as this is both efficient and reasonable. Two sequences match verbatim if all their n -grams are equal for all n , and approximately if most do for some n ( Lee et al., 2021 ) . However, this definition must align with downstream concerns in privacy, copyright, and safety—domains focused on what an LLM can reproduce as an intuitive notion of membership. Consider a setting where a long text sequence is generated verbatim by an LLM. The sequence has high entropy due to its length, which makes it very unlikely to have been generated by chance. Thus, one may intuitively conclude that this sequence was in the training data. This raises a key question: can an LLM generate a target sequence even if it was never included as an n -gram in its training data? Our findings confirm that the answer is affirmative: n -gram membership establishes a threshold dependent on n , and this threshold can be gamed . In other words, our findings show that that formally defining a robust and accurate notion of membership is challenging. Figure 1: Main setup and findings : a text sequence can be (verbatim) completed by a language model without being a n -gram “member” of its training set. Left (§ 4 ): We pretrain a model and remove extracted training dataset of length k from its training data with either exact (k=n) -gram filters or stronger approximate (k>n) -gram membership filters. We find some sequences ( \approx 40\% with exact filters or \approx 1\% with approximate filters) remain verbatim completed despite not being explicitly trained on. Right (§ 5 ): We show an LLM can be fine-tuned to verbatim complete a target unseen sequence, e.g., today’s blog post, by using adversarially constructed datasets with no n -gram overlap. We assess whether an LLM generates a target sequence by prompting it with a prefix and checking if it completes the corresponding suffix, a process we refer to as a completion test . Language models are known to complete some of their training data ( Nasr et al., 2023 ) . In our work, we first find that even after removing a set of extracted sequences from the training dataset and retraining the LLM from scratch , the retrained model can still verbatim complete 40% of them under our experimental conditions (Section 4 ). Upon investigation, we find that these removed yet still completed sequences are either de facto members of the training set (but for a different definition of membership) or lacking sufficient complexity: many examples have near duplicates, sequences with m<n -grams that are not removed, or are explained by the model’s generalization capabilities (e.g., patterns or counting). This result, however, leads to our second finding—there exists strategies for systematically gaming the n -gram membership definition. That is, there are strategies for constructing a dataset D that does not contain n -grams of a sequence x , yet, when a language model trains on D it is able to complete x verbatim. In Section 5 , we give multiple examples of such strategies, including one that has the model train on multiple m -grams of x , where m<n and, in some cases, m\ll n . Our experiments show that we can systematically force a model to complete six sequences of interest x despite these sequences not being a member of its training set per the n -gram membership definition. Figure 1 shows our setup. Our main takeaways are: 1. We find that there is high overlap between training data membership and our LLM completion test being positive. Text not in this overlap are explained by the lack of complexity or limitations in n -gram based definitions. 2. n -gram membership is limited in capturing the intuition of what constitutes a training dataset “member.” Indeed, our work shows that a model can complete sequences that are not n -gram members of its training dataset. 3. We believe that the underlying cause of this limitation is not in the choice of the distance used to compare sequences—here n -gram overlap—but rather in the fact that the membership definition fails to consider auxiliary information that the training algorithm gets access to, e.g., through pre-processing or other design choices made throughout the ML pipeline. Here, the strategies we propose to game the n -gram definition exploit this by introducing auxiliary information through the very construction of the training dataset: e.g., we cannot construct (n-1) -grams of a sequence x without knowing the entire sequence in the first place. 2 Background & Related Work Definitions of data membership. Many language model tasks require a definition of data membership. In most cases, the definition falls into versions of n -gram, or substring, overlap ( Anil et al., 2023 ; Gemini Team et al., 2023 ; Gemini Team et al., 2024 ; Gemma Team et al., 2024a ; Gemma Team et al., 2024b ; Touvron et al., 2023 ; Dubey et al., 2024 ; Zhang et al., 2024a ; Duan et al.,