VLA-Mark: A cross modal watermark for large vision-language alignment models

Shuliang Liu, Qi Zheng, Jesse Jiaxi Xu, Yibo Yan, He Geng, Aiwei Liu, Peijie Jiang, Jia Liu, Yik-Cheung Tam, Xuming Hu · 2025

Read VLA-Mark: A cross modal watermark for large vision-language alignment models by Shuliang Liu, Qi Zheng, Jesse Jiaxi Xu, Yibo Yan, He Geng, Aiwei Liu, Peijie Jiang, Jia Liu, Yik-Cheung Tam, Xuming Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-14067.txt.

Introduction

Shuliang Liu and co-authors design a watermark for vision-language models that survives where text-only marks fail. Owners of these models need proof of authorship embedded in generated text. Existing text watermarks pick tokens in biased ways that fight the image content, and fixed strategies leave key concepts exposed.

Their method, VLA-Mark, coordinates the mark with the picture at several scales: local patch affinity, global meaning, and attention patterns. An entropy-sensitive dial strengthens the mark when the model is confident and eases off when meaning is at risk. No retraining is needed. The result aims to protect model property without bending what the model says about an image.

Catalog record

CREATORShuliang Liu, Qi Zheng, Jesse Jiaxi Xu, Yibo Yan, He Geng, Aiwei Liu, Peijie Jiang, Jia Liu, Yik-Cheung Tam, Xuming Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-14067

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VLA-Mark: A cross modal watermark for large vision-language alignment models Shuliang Liu, Qi Zheng, Jesse Jiaxi Xu, Yibo Yan, He Geng, Aiwei Liu, Peijie Jiang, Jia Liu, Yik-Cheung Tam, Xuming Hu Abstract Vision-language models demand watermarking solutions that protect intellectual property without compromising multimodal coherence. Existing text watermarking methods disrupt visual-textual alignment through biased token selection and static strategies, leaving semantic-critical concepts vulnerable. We propose VLA-Mark , a vision-aligned framework that embeds detectable watermarks while preserving semantic fidelity through cross-modal coordination. Our approach integrates multiscale visual-textual alignment metrics, combining localized patch affinity, global semantic coherence, and contextual attention patterns, to guide watermark injection without model retraining. An entropy-sensitive mechanism dynamically balances watermark strength and semantic preservation, prioritizing visual grounding during low-uncertainty generation phases. Experiments show 7.4% lower PPL and 26.6% higher BLEU than conventional methods, with near-perfect detection (98.8% AUC). The framework demonstrates 96.1% attack resilience against attacks such as paraphrasing and synonym substitution, while maintaining text-visual consistency, establishing new standards for quality-preserving multimodal watermarking 1 1 1 Code will be available at https://github.com/shiningwhite-cmd/VLA-mark . 1 Introduction The emergence of vision-language aligned multimodal large models (VLAMMs) has fundamentally transformed cross-modal content generation. Pioneering architectures like LLaVA Liu et al. 2023 and Flamingo Alayrac et al. 2022 establish joint embedding spaces through cross-modal attention mechanisms, enabling unprecedented visual-linguistic synergy. These models achieve state-of-the-art performance in vision-language tasks ranging from contextual image captioning to visual commonsense reasoning, with recent extensions like Mini-Gemini Li et al. 2024b demonstrating human-level multimodal comprehension. Liu and Bu 2024 ; Yoo et al. 2024 However, their rising capability to generate semantically coherent cross-modal content urgently demands robust solutions for intellectual property protection and content authenticity . Embedding imperceptible yet detectable watermarks into LLM-generated outputs has emerged as a pivotal solution, yet existing techniques predominantly focus on unimodal scenarios. The pioneering "green list" partitioning Kirchenbauer et al. 2023 establishes fundamental watermarking frameworks through vocabulary bias induction, while subsequent improvements like unbiased probability of two partitioned lists Mao et al. 2024 and distribution-preserving strategies Wu et al. 2024 enhance quality-robustness trade-offs in text generation. However, these approaches fail to address the unique challenges of multimodal generation where visual semantics critically guide textual outputs . Current watermarking methodologies exhibit three critical limitations when applied to vision-language aligned generation. First, traditional text watermarking approaches like "green list" partitioning Kirchenbauer et al. 2023 disrupt vision-conditioned language generation by introducing vocabulary biases that contradict visual semantics - for instance, suppressing visually grounded entity mentions detected through region-based attention. Even advanced context-aware variants Ren et al. 2023 fail to account for cross-modal dependencies established through vision-language projection layers in models like BLIP-2 Li et al. 2023 . Second, static watermark allocation strategies Liang et al. 2024 ; Zhao et al. 2023 typically apply uniform injection intensities regardless of position-specific visual grounding strength, leading to disproportionate distortion of visually salient tokens. This limitation persists even in theoretically-grounded approaches Huang et al. 2023 that optimize statistical trade-offs but ignore entropy variations during cross-modal generation. Third, current methods lack explicit mechanisms to protect vision-critical semantics under text-space attacks. Random vocabulary partitioning and uniform logit manipulation render key visual concepts (e.g., objects, scene descriptors) vulnerable to adversarial paraphrasing or synonym substitution. As shown in Fig. 1 (5), conventional watermarks indiscriminately boost non-semantic tokens (green blocks) while leaving visually anchored phrases like "grassy trail" (light blue blocks) exposed to semantic erasure through token replacement attacks. This fundamentally undermines text-visual coherence and detection consistency. We resolve these challenges through VLA-Mark , the first vision-language aligned watermarking framework that achieves cross-modally coordinated, quality-preserving watermark with excellent detectability and robustness via three innovations. First, extending beyond random vocabulary splitting, our Multiscale Semantic Saliency Metrics leverage visual semantics to guide green list selection through localized patch affinity (LPA), global semantic coherence (GSC), and cross-modal contextual salience (CCS). This aligns token partitioning with image content while maintaining zero training overhead. Second, our Entropy-Regulated Partition dynamically adjusts watermark intensity based on generation uncertainty and token criticality scores, prioritizing semantic preservation in low-entropy phases while enhancing watermark strength during high-entropy generation. Third, we introduce SCT based Distribution Adjustment through vision-aligned token prioritization, where cross-modal embedding alignment and fused metrics establish hierarchical protection for Semantic Critical Tokens (SCTs) against textual perturbations. Our contributions transcend prior art through three breakthroughs: • We pioneer the first text watermarking method for vision-language models, achieving cross-modal semantic guidance through native alignment mechanisms of VLA architectures, yielding 7.4% and 26.6% average improvement (PPL↓ and BLEU↑) in textual quality with zero training overhead. • We develop an uncertainty-aware coordination mechanism that automatically adapts watermark intensity to logits entropy, breaking the preservation-detection trade-off by maintaining SOTA detection performance while enhancing generation quality. • Through dedicated SCT preservation, we establish hierarchical protection against Paraphrase, Synonym, Translate and more attacks, ensuring text-visual consistency under perturbations. 2 Methodology Our VLA-Mark framework introduces a vision-aligned watermarking method that identifies Semantic Critical Tokens (SCTs) , linguistic units strongly grounded in visual semantics guided by cross-modal embedding alignment (Sec 2.1 ) and fused multiscale metrics (Sec 2.2 ). SCTs preserve text-visual coherence by anchoring key concepts ( e.g., objects/scenes) while enabling entropy-regulated dynamic vocabulary partitioning (Sec 2.4 ): low-entropy contexts prioritize SCT retention for semantic fidelity, whereas high-entropy phases emphasize watermark strength. The method further adjusts token distributions through watermarked logit manipulation (Sec 2.5 ). This approach pioneers visual semantics as the foundation for watermark injection, contrasting traditional text-only statistical strategies, as is illustraed in Fig. 1 . For more theoretical analysis of each part, please refer to Appendix C . 2.1 Cross-Modal Aligned Embedding Figure 1: Proposed VLA-Mark framework. Vision embeddings \mathbf{H}_{v} (aligned to LLM space) and linguistic tokens \mathbf{H}_{L} extracted from LLM vocabulary \mathcal{V} compute fused multiscale metrics (LPA/GSC/CCS) to rank \mathcal{V}^{*} by visual saliency. Entropy-regulated SCT selection dynamically enhances semantic expressiveness when low entropy in logits distribution or watermark robustness when high entropy. Light blue denote SCT, which in the response is followed by conventional watermarked tokens. As demonstrated in prior research, Vision-Language Alignment (VLA) models like LLaVA Liu et al. 2023 employ a shared semantic mapping strategy where visual embeddings are projected into the text embedding space. Given a textual instruction X_{q} and visual input X_{v} , such models utilize parallel encoding streams to process multimodal inputs. The vision encoder ( e.g., SigLIP Zhai et al. 2023 or ViT-L/14 Radford et al. 2021 ) generates spatial-visual features through: \displaystyle\mathbf{Z}_{v} \displaystyle=\text{VisEnc}(X_{v})=[\mathbf{z}_{\text{cls}};\mathbf{z}_{1},...,\mathbf{z}_{P}], (1) where \mathbf{Z}_{v}\in\mathbb{R}^{(P+1)\times d_{v}} and P indicates the total number of image patch tokens augmented with a global [CLS] token. The subsequent alignment phase employs a trainable projection module f_{\theta}(\cdot) , implemented as MLP Liu et al. 2024a or generation adaptor Chen et al. 2025 , to bridge the dimensional gap between modalities: \mathbf{H}_{v}=f_{\theta}(\mathbf{Z}_{v}), (2) where f_{\theta} denotes parametric transformation that enables cross-modal compatibility while retaining original information patterns, so we get \mathbf{H}_{v}\in\mathbb{R}^{(P+1)\times d} . LLMs ( e.g., Vicuna Chiang et al. 2023 ) first tokenize input text of length S and then retrieve text embeddings \mathbf{H}_{q}\in\mathbb{R}^{S\times d} for LLM inference by querying the pretrained token embedding table, commonly referred to as the Vocabulary \mathcal{V} . We construct an embedding matrix \mathbf{H}_{L} by removing non-linguistic elements such as symbols and numbers from \mathcal{V} , where L

Read the complete text