Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking

Tianle Gu, Zongqi Wang, Kexin Huang, Yuanqi Yao, Xiangliang Zhang, Yujiu Yang, Xiuying Chen · 2025

Read Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking by Tianle Gu, Zongqi Wang, Kexin Huang, Yuanqi Yao, Xiangliang Zhang, Yujiu Yang, Xiuying Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14112.txt.

Introduction

Tianle Gu and co-authors mark text from language models in cases where the text is easy to guess in this 2025 paper. Most marks push the model toward words on a secret green list. When the next word is nearly fixed, that push can break the text's flow.

Older fixes check how unsure the model is, but they need the source model itself, which is slow and risks leaks. Their method, Invisible Entropy, judges that uncertainty without the source model. It aims to keep marks strong in flat, predictable text. The work also cuts the cost and delay of checking marks.

Catalog record

CREATORTianle Gu, Zongqi Wang, Kexin Huang, Yuanqi Yao, Xiangliang Zhang, Yujiu Yang, Xiuying Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14112

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking Tianle Gu, Zongqi Wang, Kexin Huang, Yuanqi Yao, Xiangliang Zhang, Yujiu Yang, Xiuying Chen Abstract Logit-based LLM watermarking traces and verifies AI-generated content by maintaining green and red token lists and increasing the likelihood of green tokens during generation. However, it fails in low-entropy scenarios, where predictable outputs make green token selection difficult without disrupting natural text flow. Existing approaches address this by assuming access to the original LLM to calculate entropy and selectively watermark high-entropy tokens. However, these methods face two major challenges: (1) high computational costs and detection delays due to reliance on the original LLM, and (2) potential risks of model leakage. To address these limitations, we propose Invisible Entropy (IE), a watermarking paradigm designed to enhance both safety and efficiency. Instead of relying on the original LLM, IE introduces a lightweight feature extractor and an entropy tagger to predict whether the entropy of the next token is high or low. Furthermore, based on theoretical analysis, we develop a threshold navigator that adaptively sets entropy thresholds. It identifies a threshold where the watermark ratio decreases as the green token count increases, enhancing the naturalness of the watermarked text and improving detection robustness. Experiments on HumanEval and MBPP datasets demonstrate that IE reduces parameter size by 99% while achieving performance on par with state-of-the-art methods. Our work introduces a safe and efficient paradigm for low-entropy watermarking. IE-official-repo Entropy-Tagger 1 Introduction Figure 1: Existing watermarking methods in low-entropy scenarios face safety and cost challenges, while our method addresses them efficiently and securely. Textual watermarking, which aims to embed subtle patterns in the generated text to make it detectable by algorithms but invisible to humans, is an important step towards trustworthy AI. It can be applied at various stages, including logits generation Kirchenbauer et al. 2023 , token sampling Christ et al. 2024 , and training Sun et al. 2022 ; Sun et al. 2023 ; Gu et al. 2024 . Logit-based watermarking is cost-efficient, modifying probabilities before token selection without adding training or sampling steps Liu et al. 2024 . As a pioneering work in logit-based watermarking, Kirchenbauer et al. 2023 introduced KGW, the first logit-based watermarking approach. This method partitions the vocabulary into green and red lists based on the previous token and a hash key, then boosts the logits of the green list to embed the watermark and decreases the probabilities of tokens outside this green list (red list). However, KGW fails in low-entropy scenarios where the next token is highly predictable, such as the prompt “import numpy as” almost certainly leading to“np” (entropy 0.048). If this expected token is placed in the red list, two issues may arise: (1) If the model still selects it despite the reduced probability, the unexpected inclusion of a red-list token may weaken the watermark’s detectability. (2) If the model instead picks a green-list token due to the boosted logits, it may disrupt text fluency. Similarly, if the expected token is directly placed in the green list, it may lead to a false inflation of green-list token frequency in generated text, thereby increasing the likelihood of misclassify human-written content as machine-generated. As a result, the watermark detection system becomes less reliable. To address the low-entropy problem, Lee et al. 2024 proposed SWEET, which applies watermarks only to high-entropy tokens, preserving text quality. Similarly, Lu et al. 2024 introduced EWD, which enhances detection by assigning higher weights to high-entropy tokens. However, these entropy-based watermarking methods face a critical limitation: they assume the detector has access to the original LLM to calculate entropy. This reliance on the original model introduces several challenges, as illustrated in Fig. 1 . First, providing the original model to third parties poses significant risks of model leakage, potentially leading to unintended exposure or unauthorized access Song and Raghunathan 2020 ; Duc et al. 2014 . Second, using the original LLM incurs substantial computational costs, particularly when processing large-scale datasets or running multiple detections. Using a proxy model to approximate entropy calculation is potentially feasible. SWEET replaces the original model, e.g., LLaMA2-13B Touvron et al. 2023 , with a smaller model from the same family, e.g., LLaMA2-7B, for entropy estimation. Although this practice outperforms KGW, it still suffers from significant performance degradation. While the original EWD work does not explicitly explore the use of proxy models, our experimental results in Tab. 1 show a similar trend. It is also important to note that the effectiveness of a proxy model heavily depends on its architectural similarity to the original model. Motivated by this, we attempt to train a lightweight proxy model to eliminate the dependency on the original LLM during entropy-based watermark detection. Our experiments in App. C show that regressing continuous entropy using an MLP is challenging, but reframing the task as a classification problem – determining whether the entropy of next token exceeds a given threshold – is more feasible. Considering the aforementioned issue that proxy models rely on architectural similarity to the original model, we introduce a Unified Feature Extractor that converts prefix tokens into a unified feature representation using a token translator and an embedding model, thereby ensuring compatibility across different LLMs and tokenizers. When using a fixed threshold to distinguish high- and low-entropy tokens, we observe that applying the same threshold across all samples ignores inter-sample variability. Moreover, in practical scenarios, the generator and the detector cannot share threshold, which limits the applicability of watermarking methods. To balance the naturalness of generated text and watermark detectability, we propose a sample-level entropy threshold optimization method. We evaluate our method in a representative low-entropy setting, namely the code generation task, with two widely used benchmarks: HumanEval Chen et al. 2021 and MBPP Austin et al. 2021 . Our main contributions are as follows: (1) We propose IE, a novel watermarking framework that relies on a small MLP instead of the original LLM to enable safe, efficient and accurate watermark detection. (2) We present Threshold Navigator , a low-high entropy threshold auto-optimization method that enhances detection performance not only for our framework but also for various watermarking approaches. (3) Our proposed watermarking framework, IE, which integrates the three components, achieves a 99% reduction in parameter usage while delivering state-of-the-art detection performance, showcasing its efficiency and scalability. 2 Related Work Traditional Text Watermarking typically modifies generated text to embed watermarks. Based on the granularity of these modifications, existing approaches can be categorized as format-based, lexical-based, syntactic-based, and generation-based methods. Format-based watermarking Rizzo et al. 2016 ; Brassil et al. 1995 ; Por et al. 2012 ; Sato et al. 2023 originates from image watermarking and focuses on altering the text format rather than its content, such as by adjusting text layout or using Unicode-based substitutions. Lexical-based watermarking Munyer et al. 2024 ; Ni et al. 2023 ; Yang et al. 2023 ; Yoo et al. 2023 ; Yang et al. 2022 replaces selected words with their synonyms while preserving the original sentence’s syntactic structure. However, this approach is susceptible to attacks involving random synonym replacements. To address this vulnerability, syntactic-based methods ( Atallah et al. 2001 ; Topkara et al. 2006 ; Meral et al. 2009 ) embed watermarks by

Read the complete text