Your Text Encoder Can Be An Object-Level Watermarking Controller

Naresh Kumar Devulapally, Mingzhen Huang, Vishal Asnani, Shruti Agarwal, Siwei Lyu, Vishnu Suresh Lokhande, University at Buffalo, SUNY · 2025

Read Your Text Encoder Can Be An Object-Level Watermarking Controller by Naresh Kumar Devulapally, Mingzhen Huang, Vishal Asnani, Shruti Agarwal, Siwei Lyu, Vishnu Suresh Lokhande, University at Buffalo, SUNY (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11945.txt.

Introduction

Naresh Kumar Devulapally, Mingzhen Huang, and colleagues watermark generated images at the level of single objects in a 2025 study. Invisible watermarks help owners prove that an image came from an AI system. Marking the whole picture is blunt when only one object needs protection or credit.

Their method fine-tunes only text token embeddings inside a latent diffusion model. With that small change, watermarks land in selected objects or parts of the scene. The text encoder works across many diffusion models, so the trick plugs into different generators without retraining them. Finer control makes watermarking fit mixed scenes with several owners.

Catalog record

CREATORNaresh Kumar Devulapally, Mingzhen Huang, Vishal Asnani, Shruti Agarwal, Siwei Lyu, Vishnu Suresh Lokhande, University at Buffalo, SUNY
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11945

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Your Text Encoder Can Be An Object-Level Watermarking Controller Naresh Kumar Devulapally, Mingzhen Huang, Vishal Asnani, Shruti Agarwal, Siwei Lyu, Vishnu Suresh Lokhande, University at Buffalo, SUNY Abstract Invisible watermarking of AI-generated images can help with copyright protection, enabling detection and identification of AI-generated media. In this work, we present a novel approach to watermark images of T2I Latent Diffusion Models (LDMs). By only fine-tuning text token embeddings 𝒲 ∗ \mathcal{W}_{*} , we enable watermarking in selected objects or parts of the image, offering greater flexibility compared to traditional full-image watermarking. Our method leverages the text encoder’s compatibility across various LDMs, allowing plug-and-play integration for different LDMs. Moreover, introducing the watermark early in the encoding stage improves robustness to adversarial perturbations in later stages of the pipeline. Our approach achieves 99 % 99\% bit accuracy ( 48 48 bits) with a 10 5 × 10^{5}\times reduction in model parameters, enabling efficient watermarking. Code: GitHub . Figure 1 : Text prompt controlled Object-level watermarking . Our method takes a text prompt \mathcal{P} and a watermarking token \mathcal{W}_{*} as input and integrates watermarking into text-to-image generation. (Left Image): Given a watermarking token \mathcal{W}_{*} , the user can pick any subset of tokens \{\mathcal{P}_{i},\mathcal{P}_{j},...\} within the prompt \mathcal{P} , we then watermark specific objects corresponding to these user-selected tokens within the image using cross-attention maps. The user has the flexibility to watermark full-image or watermark one or more specific objects while perfectly preserving non-watermarked regions. (Right Image): We demonstrate our method’s ability to perform object watermarking with personalization using Styled UNet and Textual Inversion. 1 Introduction As debates around copyright and intellectual property intensify, the need for effective watermarking solutions increases [ 37 ] . With generative AI models producing indistinguishable images from human-made art, maintaining authorship provenance in digital media is crucial. Efforts are underway, from legislative measures like the Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence [ 2 ] , to industry initiatives that aim to watermark all AI-generated content [ 27 ] [ 34 ] . These developments highlight the growing importance of watermarking as a key area of study. As a result, multiple Generative AI (GenAI) watermarking methods have been proposed [ 10 , 37 , 8 ] to invisibly watermark the entire image while generating the image. However, lots of current watermark methods [ 7 , 4 ] can only encode a watermark to an open-box generative model as they need to access the latent space. Moreover, similar to other invisible watermarking techniques [ 3 , 8 , 7 ] , these methods also suffer with the inherent trade-off between quality and robustness of the embedded signal. Owing to the high imperceptibility requirements for the high-quality image generation, these watermarks more often lacks robustness [ 10 , 40 ] . To improve the watermark imperceptibility, partial watermarking has been proposed to limit the watermark related changes only to selected “non-salient” objects or regions in the image [ 25 ] . In our paper, we bring such partial watermarking to text-to-image generation pipeline. Our partial watermarking is not only for improving the quality of the watermarked image, but also to give the user the flexibility to watermark only selected object of interest in the generated image without accessing the latent space. This is crucial for scenarios where the user would like to protect the unique object in the image, see Your Text Encoder Can Be An Object-Level Watermarking Controller . Specifically, we propose a token-based watermarking approach that can embed watermarks into selected objects or partial regions of an image. Unlike previous works that have explored architectural modifications, such as the addition of secret encoders [ 4 ] , distortion layers [ 7 ] , and adapters [ 6 ] , our watermarking method focuses solely on learning a watermark embedding. This allows for a prêt-à-porter style of training [ 31 ] , leveraging pre-trained models with minimal adjustments. Additionally, compared with prior works [ 4 , 6 , 7 ] , our model is blind, meaning that the entire generative model is neither modified nor requires additional training. A simple watermark token is learned by our model and can be plugged into any diffusion model for watermark generation. Previous works [ 8 , 37 ] have investigated image watermarking without altering the architecture. Although effective, they lack the convenience of prêt-à-porter training, which offers the key advantage of personalizing large models with minimal retraining while using the model’s core components as-is. In contrast, our approach enables users to apply watermarking directly at the prompt stage without modifying core model components such as the UNet or image decoder. We introduce new pseudo-tokens \bm{\mathcal{W}_{*}} into the model’s vocabulary, where the \bm{\mathcal{W}_{*}} can be used as an input text prompt to be applied on any T2I diffusion models for generation watermarked images. This also enables users to selectively watermark specific image regions by leveraging the cross-attention between the special token and visual region in the image where the watermark needs to be embedded. By integrating watermarking functionality early in the text-to-image pipeline, our approach performs in-generation watermarking, offering greater robustness against image manipulation attacks compared to post-processing techniques while improving the overall quality of the watermarked images. We make the following key contributions : 1. We introduce a novel pseudo-token with watermarking capabilities, learning a new embedding vector for it. We investigate the impact of applying this pseudo-token conditioning at different timesteps of the diffusion process, finding that timesteps closer to the VAE encoder in LDMs during forward process offer enhanced image generation quality. 2. We introduce object-level watermarking, allowing for selective watermarking with greater precision than traditional whole-image approaches. 3. We offer plug-and-play integration with various Stable Diffusion (SD) variants by embedding the watermark directly via the text encoder. Our approach achieves 99\% bit accuracy with a 10^{5}\times reduction in model parameters, enabling efficient watermarking with a throughput of 48 bits. 2 Related Works Text-Guided Image Generation Diffusion Models: Diffusion models have revolutionized text-to-image (T2I) generation, surpassing GANs in fidelity and diversity. By iteratively denoising random noise into images conditioned on text prompts, these models enable fine-grained control over generation. Frameworks like DALL-E 2, Imagen [ 14 ] , and Stable Diffusion [ 35 ] democratize high-quality image synthesis, while diffusion-based editing methods [ 13 ] enable localized modifications. However, their widespread adoption raises challenges, including copyright infringement and harmful content generation, necessitating controllable synthesis mechanisms such as watermarking. Textual Inversion for Personalized Image Generation: Textual Inversion [ 9 ] embeds visual or stylistic concepts into T2I models through learnable tokens, enabling personalization without altering model parameters. This lightweight approach has been extended to style transfer [ 38 ] . In this work, we find that training the special token not only allows for flexible concept learning but also could acts as a watermark controller. Constraint-Based Control in Text-to-Image Generation: Text-to-image generation has gained significant popularity, especially to incorporate constraints that enhance control over generated content. Common constraints include spatial or layout constraints, which dictate object placement and dimensions within the generated image to meet predefined spatial requirements [ 19 ] . Another type, multi-view consistency constraints, ensures scene consistency across perspectives, preserving layout, lighting, and depth even

Read the complete text