Watermarking Discrete Diffusion Language Models

Avi Bagchi, Akhil Bhimaraju, Moulik Choraria, Daniel Alabi, Lav R. Varshney · 2025

Read Watermarking Discrete Diffusion Language Models by Avi Bagchi, Akhil Bhimaraju, Moulik Choraria, Daniel Alabi, Lav R. Varshney (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02083.txt.

Introduction

Avi Bagchi, Akhil Bhimaraju, Moulik Choraria, Daniel Alabi, and Lav Varshney bring watermarking to a newer class of language models. Watermarks let providers mark AI-generated text so it can later be detected, but most methods target autoregressive models or image generators. Discrete diffusion language models, which build text through repeated denoising steps, had been largely overlooked despite their fast inference.

Their method applies a distribution-preserving sampling trick at every diffusion step and seeds the randomness by position in the sequence, which makes detection reliable. Tests on the LLaDA model show the watermark is detectable, and the authors prove it does not distort the model's output distribution, with false detection chances shrinking exponentially as text length grows. The scheme needs no costly tuning to deploy.

Catalog record

CREATORAvi Bagchi, Akhil Bhimaraju, Moulik Choraria, Daniel Alabi, Lav R. Varshney
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-02083

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Watermarking Discrete Diffusion Language Models Avi Bagchi, Akhil Bhimaraju, Moulik Choraria, Daniel Alabi, Lav R. Varshney Abstract Watermarking has emerged as a promising technique to track AI-generated content and differentiate it from authentic human creations. While prior work extensively studies watermarking for autoregressive large language models (LLMs) and image diffusion models, none address discrete diffusion language models, which are becoming popular due to their high inference throughput. In this paper, we introduce the first watermarking method for discrete diffusion models by applying the distribution-preserving Gumbel-max trick at every diffusion step and seeding the randomness with the sequence index to enable reliable detection. We experimentally demonstrate that our scheme is reliably detectable on state-of-the-art diffusion language models and analytically prove that it is distortion-free with an exponentially decaying probability of false detection in the token sequence length. 1 Introduction The rapid development and widespread deployment of artificial intelligence (AI) agents necessitate guarantees of authenticity and traceability ( Fairoze et al. 2025 ; Biden, Jr. 2023 ; Biden, Jr. and Harris 2023 ) . A particularly serious security concern is distinguishing AI-generated text from human-written content ( Dathathri et al. 2024 ) , as nefarious actors exploit generative AI to create deceptive content ( Tarsney 2025 ; Tang, Chuang, and Hu 2023 ) . Such misinformation pollutes the information environment and compromises the integrity of data used to train future AI models. Since the reliability of post-hoc automated detectors degrades as model quality improves ( Fairoze et al. 2025 ; Jawahar, Abdul-Mageed, and Lakshmanan 2020 ; Varshney, Keskar, and Socher 2020 ) , watermarking generative models has surfaced as a novel solution ( Aaronson and Kirchner 2022 ; Kirchenbauer et al. 2024 ) . A watermark subtly shifts the output distribution of a generative model so that it remains invisible to humans yet can be detected algorithmically ( Fairoze et al. 2025 ; Kirchenbauer et al. 2024 ; Alabi and Varshney 2025 ) . For instance, Kirchenbauer et al. 2024 propose partitioning the vocabulary into a red and green list with the model favoring the latter in sampling. Strongly biasing the sampling procedure, however, creates a precarious tradeoff between detectability and distortion ( Christ, Gunn, and Zamir 2023 ; Giboulot and Furon 2024 ) . A watermark signal that is too strong, for example, can raise the perplexity, indicating that the watermark unduly increases the entropy of the distribution of tokens relative to a baseline language model. Ideally, a watermark will satisfy the following properties. • Soundness : The detection scheme reliably identifies unwatermarked content as unwatermarked ( Fairoze et al. 2025 ; Christ, Gunn, and Zamir 2023 ; Fan et al. 2025 ) . • Completeness : The detection scheme reliably identifies watermarked content as watermarked ( Fairoze et al. 2025 ; Christ, Gunn, and Zamir 2023 ) . • Distortion-Freeness : The watermark does not significantly reduce the quality of the text ( Fairoze et al. 2025 ; Christ, Gunn, and Zamir 2023 ; Giboulot and Furon 2024 ) . Completeness and distortion-freeness are often inversely related. • Robustness : The watermark is still detectable following bounded modifications to the text ( Fairoze et al. 2025 ; Giboulot and Furon 2024 ; Li et al. 2025b ; Kuditipudi et al. 2024 ; Fan et al. 2025 ) . A watermark may satisfy the aforementioned properties for one architecture yet fail miserably for another, as successful schemes often exploit idiosyncrasies in the sampling procedure ( Fernandez 2025 ) . Thus, as architectures evolve, so should watermarks . Researchers have proposed watermarks for autoregressive large language models (LLMs) ( Aaronson and Kirchner 2022 ; Kirchenbauer et al. 2024 ) and image diffusion models ( Wen et al. 2023 ) , but none for discrete diffusion—a notable gap given their rapid growth in both research and commercial use ( Google DeepMind 2025 ; Inception Labs et al. 2025 ) . Our contributions are as follows. 1. We introduce the first watermarking method for discrete diffusion models by applying the distribution-preserving Gumbel-max trick at every diffusion step and seeding the randomness with the sequence index to enable reliable detection. 2. We analytically prove that the false detection probability decays exponentially with the number of generated tokens and that our watermark leaves the token sampling distribution unchanged. 3. We implement our watermark on the state-of-the-art Language Diffusion Model LLaDA ( Nie et al. 2025a ) and achieve high completeness and soundness. While green-list methods significantly decrease LLaDA performance on math and logic benchmarks, our Gumbel-max technique entirely preserves both benchmark scores and perplexity. 2 Related Work Discrete Diffusion Models: Diffusion models gradually corrupt data into a simple distribution and then learn a reverse process to reconstruct the original data ( Sohl-Dickstein et al. 2015 ; Ho, Jain, and Abbeel 2020 ) . Although earlier work focused on learning continuous data, diffusion model methods have been adapted for categorical domains. Unlike their autoregressive counterparts that capture left-to-right dependencies, discrete diffusion models generate tokens in parallel, enabling faster inference ( Wang et al. 2025 ) , greater controllability ( Schiff et al. 2025 ) , and enhanced comprehension of global patterns ( Hu et al. 2021 ) . While the first discrete diffusion models mapped discrete tokens into continuous space before applying traditional continuous diffusion ( Li et al. 2022 ; Gong et al. 2023 ; Han, Kumar, and Tsvetkov 2023 ) , more recent models operate directly on the discrete state space ( Hoogeboom et al. 2021 ; Austin et al. 2021 ; Meng et al. 2023 ; Lou, Meng, and Ermon 2024 ; Nie et al. 2025a ) . Inception Labs’ models empirically validate discrete diffusion at scale across text and multimodal domains ( Inception Labs et al. 2025 ; Li et al. 2025a ) . Watermarking Generative Models: In continuous settings (e.g. images, audio, etc.), watermarking is well-studied as one can exploit known signal processing properties to achieve a robust fingerprint ( Wen et al. 2023 ; Katzenbeisser and Petitcolas 2000 ) . For discrete diffusion, however, modifications similar to Wen et al. 2023 not only yield completely incomprehensible outputs but are also undetectable. Kirchenbauer et al. 2024 introduce a watermark for models generating discrete data (i.e., autoregressive LLMs) which biases the sampling procedure to favor a subset of the vocabulary (i.e., the “green list”) with the hope that there is a statistically greater number of tokens from the green list in the output. While Kirchenbauer et al. 2024 propose an arbitrary threshold for watermark detection, Li et al. 2025c introduce a hypothesis-testing framework to systematically evaluate and compare different watermark detection methods. He et al. 2025 extends this perspective by jointly optimizing both watermark embedding and detection, detailing the trade-off between detectability and distortion—a balance we also empirically reveal in this paper. 3 Large Language Diffusion Models In this section, we briefly explain the LLaDA model introduced by Nie et al. 2025a as an exemplary discrete diffusion model and define the mathematical abstractions we need to develop our watermarking scheme. Consider a diffusion language model with vocabulary \mathcal{V} that generates sequences of maximum length d . Let M\in\mathcal{V} be the special mask token for diffusion. The model starts at t=1 with a sequence

Read the complete text