SWAP: Towards Copyright Auditing of Soft Prompts via Sequential Watermarking

Wenyuan Yang, Yichen Sun, Changzheng Chen, Zhixuan Chu, Jiaheng Zhang, Yiming Li, Dacheng Tao · 2025

Read SWAP: Towards Copyright Auditing of Soft Prompts via Sequential Watermarking by Wenyuan Yang, Yichen Sun, Changzheng Chen, Zhixuan Chu, Jiaheng Zhang, Yiming Li, Dacheng Tao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04711.txt.

Introduction

Wenyuan Yang leads seven authors with SWAP, a way to audit copyright in soft prompts. Soft prompts are small trained modules that adapt big vision-language models like CLIP. Owners need to prove when a suspect model quietly reuses their prompt.

Standard ownership checks fail here for structural reasons, the authors show. Non-intrusive audits raise false alarms when independent models share data patterns. Backdoor-style intrusive methods also break under prompt learning's traits. SWAP instead uses sequential watermarking designed for this setting. Reliable auditing could let prompt creators share their work without losing control of it.

Catalog record

CREATORWenyuan Yang, Yichen Sun, Changzheng Chen, Zhixuan Chu, Jiaheng Zhang, Yiming Li, Dacheng Tao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04711

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SWAP: Towards Copyright Auditing of Soft Prompts via Sequential Watermarking Wenyuan Yang, Yichen Sun, Changzheng Chen, Zhixuan Chu, Jiaheng Zhang, Yiming Li, Dacheng Tao Abstract Large-scale vision-language models, especially CLIP, have demonstrated remarkable performance across diverse downstream tasks. Soft prompts, as carefully crafted modules that efficiently adapt vision–language models to specific tasks, necessitate effective copyright protection. In this paper, we investigate model copyright protection by auditing whether suspicious third-party models incorporate protected soft prompts. While this can be viewed as a special case of model ownership auditing, our analysis shows that existing techniques are ineffective due to prompt learning’s unique characteristics. Non-intrusive auditing is inherently prone to false positives when independent models share similar data distributions with victim models. Intrusive approaches also fail: backdoor methods designed for CLIP cannot embed functional triggers, while extending traditional DNN backdoor techniques to prompt learning suffers from harmfulness and ambiguity challenges. We find that these failures in intrusive auditing stem from the same fundamental reason: watermarking operates within the same decision space as the primary task yet pursues opposing objectives. Motivated by these findings, we propose sequential watermarking for soft prompts (SWAP), which implants watermarks into a different and more complex space. SWAP encodes watermarks through a specific order of defender-specified out-of-distribution classes, inspired by the zero-shot prediction capability of CLIP. This watermark, which is embedded in a more complex space, keeps the original prediction label unchanged, making it less opposed to the primary task. We further design a hypothesis-test-guided verification protocol for SWAP and provide theoretical analyses of success conditions. Extensive experiments on 11 datasets demonstrate SWAP’s effectiveness, harmlessness, and robustness against potential adaptive attacks. keywords Model watermarking, Copyright protection, Prompt tuning, Vision-language model † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction Vision-language models (VLMs), especially contrastive language-image pretraining (CLIP) ( 48 ) , have demonstrated remarkable capabilities across various tasks, including semantic segmentation, cross-modal retrieval, vision question answering, action recognition, and unsupervised semantic segmentation ( 74 ; 56 ; 59 ; 60 ) , with their ability to align visual and textual representations for effective zero-shot transfer learning. In particular, researchers have increasingly adopted soft prompting techniques ( 72 ; 27 ; 28 ; 5 ) to enhance CLIP’s performance through learned continuous vector representations that guide the model toward specific tasks without parameter modification. These learned soft prompts can be used as plug-and-play components, enabling their rapid adaptation to new domains and improving their performance on specialized tasks ( 16 ; 70 ; 26 ) . In general, the training of these soft prompts requires substantial computational resources and domain-specific data ( e.g . , medical records or artworks), making them valuable intellectual property assets. In many cases, these soft prompts are public ( 72 ; 73 ; 27 ) . For example, developers may upload them to open-sourced platforms ( e.g . , GitHub and Hugging Face) and allow their usage for educational or research purposes; developers may also sell them and require that they cannot be sold again without permission. However, intellectual property concerns also arise due to their public nature, where adversaries may exploit these prompts for impermissible commercial purposes ( 33 ) . Currently, model ownership auditing stands as the most established and effective approach for safeguarding the copyright of publicly released models. Existing model ownership auditing methods can be categorized into two classes: non-intrusive auditing and intrusive auditing. Non-intrusive auditing methods ( 42 ; 47 ; 50 ) extract inherent yet distinctive characteristics of a model, such as class-wise distances, adversarial perturbation responses, or decision trajectories, and use them to construct a unique fingerprint. This fingerprint serves as an identifier to determine whether a suspicious model preserves the same underlying learned patterns as the original one. Intrusive auditing methods ( 61 ; 67 ; 14 ) , in contrast, rely on the intrusive injection of a developer-specific identifier into the model, thereby embedding a unique and verifiable signature into its behavior to enable direct ownership verification. A widely adopted intrusive approach is backdoor-based watermarking ( 44 ; 40 ; 38 ) , which introduces specific external or artificial features during training. This process typically consists of two stages: model watermarking and ownership verification. In the watermarking stage, developers embed distinctive patterns into the model such that any unauthorized derivative inevitably inherits them. In the subsequent ownership verification stage, defenders determine whether a suspicious third-party model infringes on the original by extracting its potential watermark and comparing it against the owner-specified reference. Notably, ownership verification is often performed in a black-box setting 18 ; 32 ; 51 , where the defender can only interact with the model through queries and observe its outputs, without access to source files or intermediate states ( e.g . , gradients). This constraint arises because third-party commercial models are commonly deployed as services accessible solely via APIs. Arguably, the copyright protection of soft prompts can be regarded as a special case of model ownership auditing, since their contribution is primarily manifested in the inference behavior of the specialized models they augment. However, unlike conventional model ownership auditing method, which typically treats the entire model as the verification target, prompt ownership verification introduces new challenges due to the distinctive nature of prompt learning. The most critical difficulty lies in the extremely limited parameter space of soft prompts (typically less than 0.1% of the overall CLIP model), as the backbone architecture of CLIP is usually left intact to preserve its generalization ability in prompt learning. In this paper, we show that existing model ownership auditing techniques fail to address these unique challenges. We first reveal that non-intrusive auditing methods ( i.e . , fingerprinting) are susceptible to false positives, as models trained on datasets with similar distributions tend to converge toward analogous intrinsic features, consistent with findings in traditional image classification 34 . This problem is exacerbated in CLIP soft prompts, where the small-scale modifications leave the model’s large-scale features essentially unchanged. Given these fundamental limitations, this paper turns to intrusive auditing, particularly backdoor-based watermarking, as a more viable path. Unfortunately, developing an effective backdoor-based auditing scheme for protecting soft prompts in CLIP models is non-trivial. Directly applying existing backdoor attacks against CLIP models, or adapting existing backdoor watermarking methods designed for traditional DNNs, both encounter inherent limitations. Specifically, the direct application of existing backdoor attacks against CLIP models to soft prompts fails to embed the watermark. We argue that this is mainly because these methods are designed to modify a massive number of model parameters to succeed. In contrast, soft prompts represent a parameter-efficient module comprising only a tiny fraction of the total CLIP parameters. The small parameter space are insufficient to make these poisoning-based methods (with limited poisoned samples) effective. Furthermore, by adapting existing backdoor methods for traditional deep neural networks, we propose a backdoor-based watermarking scheme (dubbed ‘BWAP’) that embeds misclassification behaviors toward specific verification classes into soft prompts as showin in Figure 1 (a). However, we reveal that BWAP suffers from two crucial limitations: harmfulness and ambiguity 1 1 1 Different from 12 , the ‘ambiguity’ is not defined as generating the potential watermark pattern of

Read the complete text