Rethinking Forgery Attacks on Semantic Watermarks in Black-Box Settings: A Geometric Distortion Perspective

Cheng-Yi Lee, Yichi Zhang, Yuchen Yang, Chun-Shien Lu, Jun-Cheng Chen · 2026

Read Rethinking Forgery Attacks on Semantic Watermarks in Black-Box Settings: A Geometric Distortion Perspective by Cheng-Yi Lee, Yichi Zhang, Yuchen Yang, Chun-Shien Lu, Jun-Cheng Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29807.txt.

Introduction

Lee and four colleagues rethink forgery attacks on semantic watermarks. These watermarks hide information in the starting noise of a latent diffusion model, an image generator that works in a compressed space, so ownership can later be verified.

Attackers without access to the target model use a stand-in model to forge watermarks. The authors analyse the attack through rate-distortion theory, which trades accuracy against size, and show an irreducible distortion floor. Mismatches between the stand-in and target models cap how faithful a forgery can be. The errors form structured drift and deformation rather than random noise. Building on this, they propose a scheme-agnostic detector that flags forged samples before watermark verification, and it holds up in tests.

Catalog record

CREATORCheng-Yi Lee, Yichi Zhang, Yuchen Yang, Chun-Shien Lu, Jun-Cheng Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29807

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Forgery Attacks on Semantic Watermarks in Black-Box Settings: A Geometric Distortion Perspective Cheng-Yi Lee, Yichi Zhang, Yuchen Yang, Chun-Shien Lu, Jun-Cheng Chen Abstract Recent studies have shown that semantic watermarks, which embed information into the initial noise of latent diffusion models (LDMs), are vulnerable to black-box forgery attacks. However, existing methods primarily rely on empirical evidence and lack a rigorous theoretical understanding of the conditions under which such attacks succeed or fail. To bridge this gap, we rethink the nature of such attacks through the lens of rate-distortion in the latent space. Our analysis identifies an irreducible distortion floor due to structural mismatches between proxy and target models, which fundamentally limits the fidelity of forged watermarks. We further characterize this distortion as structured geometric deviations on the latent manifold, in the form of global drift and local deformation rather than stochastic noise. Leveraging these insights, we propose a scheme-agnostic detection method that distinguishes forged samples before watermark verification. Extensive experiments demonstrate the effectiveness of our method across diverse black-box scenarios, while preserving robustness to common distortions. 1 Introduction The increasing proliferation of AI-generated content (AIGC) ( 11 ) has attracted widespread interest across various fields and contributed to substantial commercial value ( 4 ; 38 ) . In visual content generation, diffusion models ( 23 ; 47 ; 42 ) allow individuals from diverse backgrounds to produce high-quality images with minimal effort. However, this advancement has raised concerns regarding the erosion of trust in digital media ( 21 ) and the dissemination of misinformation ( 26 ) . For instance, deepfakes ( 52 ) , highly realistic AI-generated media, have been used to perpetrate fraud, damage personal reputations, and spread disinformation. In response, governments ( 6 ; 31 ; 49 ) have begun mandating that companies ensure the detectability and traceability of generated images or their underlying models, as well as the identification of responsible users. Generative image watermarking has emerged as a promising paradigm to address these challenges. By injecting watermarks into generated images, the service provider (SP) can enable reliable detection and source attribution through watermark extraction. However, post-hoc watermarking schemes ( 17 ; 10 ; 44 ) alter the data distribution and degrade visual fidelity, regardless of whether operating in the spatial ( 34 ) or frequency domain ( 1 ) . To balance utility and robustness, the semantic watermarks ( 51 ; 54 ; 22 ) modify the initial latent noise to embed a predefined pattern, which can later be recovered through inversion of the denoising process. This design enables straightforward deployment into existing diffusion models, achieving greater robustness against diverse image transformations and adversarial attacks. Figure 1 : Illustration of the black-box forgery attack. An adversary uses a proxy model to invert a watermarked image into a latent representation and generates a forged image via different strategies, which preserves the service provider’s watermark while falsifying content provenance. Despite these advantages, recent studies ( 39 ) reveal that an adversary can perform a watermark forgery attack using only black-box access to the SP’s model. Fig. 1 illustrates this attack. The adversary takes a watermarked image generated by an SP, inverts it into the latent space using a proxy diffusion model, and regenerates a new image such that the SP’s watermark is preserved while the image content is no longer produced by the SP. Such an attack undermines trust in the watermarking system by falsely attributing image provenance to an SP and wrongly accusing regular users of originating harmful content. However, prior work primarily demonstrates such attacks empirically, with limited insight into the conditions under which black-box forgery succeeds or fails. As a result, it remains unclear how much security semantic watermarking systems can provide in practice, or how to design defenses that generalize beyond specific attack implementations. In this paper, we revisit the feasibility of black-box forgery attacks on semantic watermarks.We posit that the success of such attacks is constrained by distortions introduced in the latent space when the adversary relies on a mismatched proxy model. To formalize this intuition, we model black-box forgery as a rate–distortion problem, where the adversary must trade off successful watermark information transfer (rate) against the preservation of latent quality (distortion). Under this framework, we show that proxy–target model mismatch induces an irreducible distortion floor (see Fig. 2 ), which fundamentally limits the adversary’s ability to achieve high-fidelity forgery. Crucially, we find that this distortion is not stochastic noise, but manifests as structured geometric deviations from the intrinsic latent manifold, characterized by global drift and local deformation. By leveraging these geometric differences, forged samples can be detected as a pre-verification step, without requiring any modification to existing watermarking schemes. Experimental results validate the effectiveness of our approach across a wide range of black-box forgery settings. Our contributions can be summarized as follows: \bullet To the best of our knowledge, we are the first to formalize black-box forgery attacks within a rate–distortion framework, identifying an irreducible distortion floor that fundamentally constrains the adversary’s capability. \bullet We characterize forgery-induced distortion as structured geometric deviations on the intrinsic latent manifold, manifesting as global drift and local deformation. \bullet We propose a scheme-agnostic detection based on these geometric findings. Extensive evaluations demonstrate its effectiveness under diverse black-box forgery scenarios. 2 Related Works Semantic Watermarking. Semantic watermarks ( 51 ; 54 ; 30 ) embed or map a specific, recoverable structure into the starting latent, from which the diffusion process generates watermarked images. During extraction, the denoising inversion process is applied to the watermarked image to recover the corresponding watermarked latent. This approach is highly effective because it is easy to implement without additional training. For example, Tree-Ring (TR) ( 51 ) embeds circular patterns into the frequency domain of the latent z^{(w)}_{T} . For detection, it verifies the pattern by checking if the frequency representation of \hat{z}^{(w)}_{T} is sufficiently close to the original pattern. Gaussian Shading (GS) ( 54 ) combines stream cipher encryption with distribution-preserving sampling to ensure that watermarked images follow the same distribution as non-watermarked ones. During verification, this process is inverted to recover a bit string, which is compared against registered keys. However, recent works ( 39 ) show that an adversary can readily forge a watermarked pattern using any arbitrary proxy model through reprompting and imprint attacks. Further details on semantic watermarks are provided in Sec. A.1 . Rate-Distortion Theory. Shannon first explored the essential balance between the minimum amount of information (rate) required to represent a source and the distortion that arises when the data is reconstructed ( 46 ; 45 ) . By establishing theoretical limits on compression performance, rate–distortion (RD) theory guides the design of practical source coding schemes ( 2 ) and enables evaluation of their capabilities. Recent studies ( 7 ; 8 ; 56 ) have extended this theory to include perceptual quality, revealing a three-way trade-off among rate, distortion, and perception. In this work, we leverage RD theory to characterize latent distortions arising from model mismatch. Our analysis reveals that forged latents exhibit measurable geometric drift, which aids in the separation of forged samples. Figure 2 : Latent distortion under model mismatch. Discrepancies between the target model ( \Theta ) and the proxy

Read the complete text