Multi-use LLM Watermarking and the False Detection Problem
Zihao Fu Oxford Internet Institute University of Oxford Oxford, UK [email protected] Chris Russell Oxford Internet Institute University of Oxford Oxford, UK [email protected] · 2025
Read Multi-use LLM Watermarking and the False Detection Problem by Zihao Fu Oxford Internet Institute University of Oxford Oxford, UK [email protected] Chris Russell Oxford Internet Institute University of Oxford Oxford, UK [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15975.txt.
Introduction
Zihao Fu and Chris Russell study a flaw in watermarking text from large language models. Watermarks can flag machine written text, or carry a key that names which user produced it. Using one embedding for both jobs creates false detections as the user base grows. Their 2025 paper traces why that happens.
Theory shows unwatermarked text grows more likely to be misread as watermarked as user capacity rises. The authors then propose dual watermarking, which encodes the detection mark and the user mark jointly. Experiments back the theory and show far fewer false positives. Detection accuracy stayed high under the new scheme, they report. The problem grows worse as more users join a system.
Catalog record
| CREATOR | Zihao Fu Oxford Internet Institute University of Oxford Oxford, UK [email protected] Chris Russell Oxford Internet Institute University of Oxford Oxford, UK [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15975
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-use LLM Watermarking and the False Detection Problem Zihao Fu Oxford Internet Institute University of Oxford Oxford, UK [email protected] Chris Russell Oxford Internet Institute University of Oxford Oxford, UK [email protected] Abstract Digital watermarking is a promising solution for mitigating some of the risks arising from the misuse of automatically generated text. These approaches either embed non-specific watermarks to allow for the detection of any text generated by a particular sampler, or embed specific keys that allow the identification of the LLM user. However, simultaneously using the same embedding for both detection and user identification leads to a false detection problem, whereby, as user capacity grows, unwatermarked text is increasingly likely to be falsely detected as watermarked. Through theoretical analysis, we identify the underlying causes of this phenomenon. Building on these insights, we propose Dual Watermarking which jointly encodes detection and identification watermarks into generated text, significantly reducing false positives while maintaining high detection accuracy. Our experimental results validate our theoretical findings and demonstrate the effectiveness of our approach. 1 Introduction Large Language Models (LLMs) ( Radford et al. 2018 ; Radford et al. 2019 ; Brown et al. 2020 ; Achiam et al. 2023 ; Touvron et al. 2023a ; Touvron et al. 2023b ) have emerged as the dominant technology across a wide range of natural language processing tasks. While these models have significantly advanced the field, their misuse has raised numerous ethical concerns. Of particular concern is the use of LLMs to impersonate human text, and to generate text that appears to represent a sincere effort to engage, but is in fact automatically generated. Example use cases include the automatic generation of homework; of scientific papers, grants, and reviews; and the automatic generation of spam and astroturfing ( Wachter et al. 2024 ; Yang et al. 2023 ; Nikiforovskaya et al. 2020 ) . The primary concern here is that such automatically generated text can flood the environment, making it impossible to identify sincere texts that are worth reading and responding to. In response to these harms, researchers proposed the use of digital watermarking. These watermarking techniques can be used for the detection of LLM generated text, i.e., confirming that a text was automatically generated by an LLM, or for identification , identifying a particular user id or key associated with the generation of the text. In detection , Aaronson and Kirchner 2023 ; Fernandez et al. 2023 proposed distribution-based watermarking which alter token probabilities, while Kirchenbauer et al. 2023a ; Yoo et al. 2023a introduced dictionary-based approaches that split the dictionary into parts. Then, these approaches detect the watermark with statistical tests. In identification , Fernandez et al. 2023 ; Yoo et al. 2023a demonstrated digital watermarking can identify individual users. Identification-based watermarking carries with it the risk of privacy violations. For example, if LLMs are used as part of a pseudoanonymization process to rephrase text, it could allow the text to be traced back to the account that rephrased the text. However, if appropriately disclosed, it could replace more intrusive governance measures. Such watermarking would allow LLM service providers to identify and shut down accounts used for generating spam without requiring the monitoring of every customer’s API call. Detection watermarks can be formalized as binary labels that signal a text is watermarked, while identification watermarks encode specific information, such as user identity, as an integer key. Both distribution and dictionary-based watermarking can simultaneously incorporate both types of watermarks ( Fernandez et al. 2023 ; Yoo et al. 2023a ) . This enables the detection of whether a given text is generated by an LLM while simultaneously tracking user IDs. All existing systems that simultaneously handle both the detection and identification use the same information, and share the same shortcoming. As the key capacity (e.g. maximum supported user count) increases, unwatermarked text is increasingly likely to be wrongly detected as watermarked. We refer to this as the false detection problem . This issue is evident in both widely used approaches: distribution-based and dictionary-based watermarking methods Fernandez et al. 2023 ; Yoo et al. 2023a . Both methods encode the entire text using an identification watermark and compare the maximum score among all candidate IDs against a threshold to get the detection watermark. However, as the number of candidates increases, this maximum operation distorted the test statistic distribution, resulting in more unwatermarked text being falsely detected as watermarked provided the False Negative Ratio (FNR) is held constant. As discussed by Liu et al. 2023a ; Giboulot and Teddy 2024 , such false positives are considerably more critical than false negatives, as erroneously identifying human-generated texts as watermarked can result in more severe adverse consequences, including accusations of cheating in a scholarly context, and the wrongful suspension or shutdown of user accounts. To understand why this problem occurs, we demonstrate, both theoretically and empirically, that as the identity number increases, the false detection bound increases exponentially. This rapid increase causes detection to fail, even within a relatively small key capacity, limiting our ability to embed richer information while still using watermarks for the original purpose of confirming that a text was LLM generated. To address this issue, we propose a Dual Watermark (DW) scheme to mitigate the problem of false detections. This scheme encodes the detection watermark and the identification watermark into distinct parts of the generated text. A hash decision function determines if the current step encodes a detection or identification watermark. When detecting the watermark, we reuse the same hash decision function in conjunction with two statistics for the detection and the identification watermark respectively. This can be seamlessly incorporated into both distribution-based and dictionary-based watermarking methods. We also propose an extended Hybrid Dual Watermark (HDW) strategy, which simultaneously integrates the statistics for the indicator and the identification watermark, and compare the two approaches. We conduct a theoretical analysis of the false positive error bounds associated with the distribution-based and dictionary-based watermarking methods, as well as our proposed DW method. Our analysis shows that at a particular FNR, the False Positive Ratio (FPR) grows uncontrollably for the distribution-based method ( Section 4.1 ) and the dictionary-based method ( Appendix A.8 ), when the key space – corresponding to the number of user identities – is large or the text length is short. We also present theoretical analysis of our DW method, demonstrating that it significantly outperforms methods that reutilize the same statistic for the identification watermark to calculate the detection watermark (referred to as Full Key Encoding (FKE) in our paper). Both our theoretical and empirical findings indicate that our approach substantially alleviates the false detection problem at any FNR. Our contributions are as follows: (1) We uncover the false detection problem in LLM watermarking and conduct a theoretical and empirical analysis to investigate it; (2) We propose an analysis of the false detection rate, and illustrate the potential severity of this problem under specific conditions, such as variations in text length, key space size, and other factors; (3) We introduce two new methods DW, and HDW to reduce the false detection rate, while preserving a high true positives rate. 2 Related Works As mentioned, existing watermarking methods can be categorized into two types: detection watermarks and