SAPL: Semantic-Agnostic Prompt Learning in CLIP for Weakly Supervised Image Manipulation Localization

Xinghao Wang, Changtao Miao, Dianmo Sheng, Tao Gong, Qi Chu, Nenghai Yu, Quanchen Zou, Deyue Zhang, Xiangzheng Zhang · 2026

Read SAPL: Semantic-Agnostic Prompt Learning in CLIP for Weakly Supervised Image Manipulation Localization by Xinghao Wang, Changtao Miao, Dianmo Sheng, Tao Gong, Qi Chu, Nenghai Yu, Quanchen Zou, Deyue Zhang, Xiangzheng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06222.txt.

Introduction

Finding exactly where an image was manipulated usually needs costly pixel-level labels for training. Xinghao Wang and 8 co-authors propose a weakly supervised alternative in a 2026 paper, learning from simple image-level labels only. Their observation is that features change much more at manipulation boundaries than inside regions.

Their SAPL method teaches the CLIP model prompts that encode boundary cues rather than object meaning. One module builds text prompts from edge-enhanced image features through attention. Another contrasts genuine edge patches against manipulated ones at several levels to sharpen discrimination. The final similarity map highlights tampered regions. Across several public benchmarks, SAPL reached state-of-the-art localization performance despite the weaker supervision.

Catalog record

CREATORXinghao Wang, Changtao Miao, Dianmo Sheng, Tao Gong, Qi Chu, Nenghai Yu, Quanchen Zou, Deyue Zhang, Xiangzheng Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-06222

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAPL: Semantic-Agnostic Prompt Learning in CLIP for Weakly Supervised Image Manipulation Localization Xinghao Wang, Changtao Miao, Dianmo Sheng, Tao Gong, Qi Chu, Nenghai Yu, Quanchen Zou, Deyue Zhang, Xiangzheng Zhang Abstract Malicious image manipulation threatens public safety and requires efficient localization methods. Existing approaches depend on costly pixel-level annotations which make training expensive. Existing weakly supervised methods rely only on image-level binary labels and focus on global classification, often overlooking local edge cues that are critical for precise localization. We observe that feature variations at manipulated boundaries are substantially larger than in interior regions. To address this gap, we propose Semantic-Agnostic Prompt Learning (SAPL) in CLIP, which learns text prompts that intentionally encode non-semantic, boundary-centric cues so that CLIP’s multimodal similarity highlights manipulation edges rather than high-level object semantics. SAPL combines two complementary modules Edge-aware Contextual Prompt Learning (ECPL) and Hierarchical Edge Contrastive Learning (HECL) to exploit edge information in both textual and visual spaces. The proposed ECPL leverages edge-enhanced image features to generate learnable textual prompts via an attention mechanism, embedding semantic-irrelevant information into text features, to guide CLIP focusing on manipulation edges. The proposed HECL extract genuine and manipulated edge patches, and utilize contrastive learning to boost the discrimination between genuine edge patches and manipulated edge patches. Finally, we predict the manipulated regions from the similarity map after processing. Extensive experiments on multiple public benchmarks demonstrate that SAPL significantly outperforms existing approaches, achieving state‑of‑the‑art localization performance. 1 Introduction Digital images pervade modern information dissemination. Malicious image manipulations that spread fabricated news, enable academic fraud and support illicit activities pose significant threats to societal security. As a result, accurate detection and precise localization of manipulated regions has emerged as a critical research challenge. Figure 1 : Subtle manipulation artifacts can be observed along the boundaries of manipulated regions. Our approach produces sharper heatmaps around manipulation boundaries. Early techniques combined pixel-level cues including noise features boundary information and color inconsistencies to identify manipulated regions. In recent years fully supervised methods 3 ; 12 ; 28 ; 22 have dominated. CAT-Net 15 learns manipulation artifacts in both RGB and DCT domains. MVSS‑Net 6 uses multi scale noise and boundary features for precise localization. Mantra‑Net 35 employs fine‑grained classification and boundary attention to precisely delineate manipulated edges. PROMPT‑IML 17 aligns and fuses semantic and high‑frequency features from pre‑trained vision models for cross‑dataset generalization. Sparse-ViT 29 utilizes sparse self‑attention to extract non‑semantic cues, reducing parameter count while enhancing robustness. Re-MTKD 37 presents a multi teacher knowledge distillation framework that embeds both shared and unique manipulation patterns in the student model. Despite strong performance on training sets, these methods often suffer from poor generalization to unseen manipulation types, and the high cost of pixel‑level annotations hinders large‑scale dataset construction. In contrast, weakly supervised methods 43 rely solely on image‑level binary labels, obviating the need for pixel masks while retaining localization accuracy and generalization capability. Compared to unsupervised 39 or semi‑supervised schemes, weak supervision offers more stable optimization and requires more readily available annotations. For instance, WSCL 38 ; 24 ; 1 achieves localization through multi‑noise view branches and cross‑patch self‑consistency. SO‑WSL 44 refines pseudo‑labels via iterative optimization. M 2 RL‑Net 16 combines patch‑level self‑consistency and feature‑level contrastive learning across multiple noise branches. Despite achieving significant results, these approaches often fail to systematically integrate edge information, which is essential for precise localization. Figure 2 : Four low-level statistics (mean local variance, mean gradient magnitude, intensity skewness and excess kurtosis) computed over four region types (manipulated edge / manipulated inner / authentic edge / authentic inner) on CASIAv2. Manipulated edges consistently show higher noise and gradient and altered higher-order statistics (mean ± std), supporting the motivation to focus on boundary cues. As established in image manipulation analysis 35 ; 44 ; 23 ; 2 ; 33 , common manipulation types such as splicing 5 , copy-move 27 , and inpainting disrupt the local consistency of natural images. Interior regions usually preserve intrinsic cues such as CFA artifacts and JPEG traces, while manipulation boundaries act as interfaces between distinct image segments. This discontinuity results in variations of noise distribution, illumination, and texture gradients. Consequently, edges become the most reliable indicators of manipulation traces. To verify the importance of edge features, we conduct a statistical analysis of manipulation trace distributions as shown in Fig. 2 . Specifically, we compute the mean local variance of grayscale values, mean gradient magnitude (Sobel), intensity skewness, and excess kurtosis in four regions: manipulated edges, authentic edges, manipulated interiors, and authentic interiors. These metrics respectively capture the local noise fluctuation, structural strength, and higher-order statistical characteristics of each region. The results show that manipulated edges exhibit much stronger brightness variations than authentic edges, substantially higher gradient magnitudes, and noticeably altered skewness and kurtosis distributions, while the interior regions of both manipulated and authentic areas share similar local statistics. Existing weakly supervised methods 38 ; 44 ; 16 often overlook low level cues such as edge and noise patterns, which limits their ability to learn boundary discriminative features without pixel level supervision. Although CLIP 25 is powerful for semantic alignment, it is even less sensitive to these low level signals. This observation motivates us to exploit edge information as a fundamental structural signal that directly reflects inconsistencies caused by manipulation. However, effectively utilizing edge information under weak supervision is highly challenging because pixel level annotations are unavailable to explicitly guide boundary learning. To address this issue, we introduce Semantic-Agnostic Prompt Learning (SAPL), a weakly-supervised framework that explicitly learns semantic-agnostic text prompts to inject boundary-centric cues into CLIP 25 , thereby steering multimodal alignment away from object semantics toward manipulation artifacts. Specifically, we propose an Edge-aware Contextual Prompt Learning module (ECPL) built on CLIP 25 . ECPL constructs learnable prompt templates for the real and manipulated class and personalizes these prompts by embedding per image edge cues derived from patch level features and edge maps. This strategy compels the model to attend to non semantic boundary discrepancies thereby enhancing localization precision and cross dataset generalization without reliance on pixel-level masks. To further strengthen feature discrimination, we introduce a Hierarchical Edge Contrastive Learning module (HECL). HECL assumes that edge features of the same class should cluster closely in the latent space. It selects high confidence edge patches across multiple encoder layers forms positive and negative pairs based on edge cues and applies contrastive learning to enforce separation between authentic and manipulated boundaries. The combination of ECPL and HECL enables SAPL to efficiently exploit edge information within CLIP 25 , achieving precise and robust image manipulation localization under weak supervision. Experiments show that our method achieves state-of-the-art performance in both manipulation detection and localization. Our main contributions are as follows: Figure 3 : The SAPL architecture. SAPL implements semantic-agnostic prompt learning via two complementary modules: Edge-aware Contextual Prompt Learning (ECPL) and Hierarchical Edge Contrastive Learning (HECL). During training, SAPL extracts patch features and soft edge maps, applies ECPL for prompt generation and HECL for edge contrastive learning, and optimizes classification plus contrastive losses. At inference, it uses patch–text similarity to produce localization maps and decide the class. • We design Edge-aware Contextual Prompt Learning, which embeds image-specific edge cues into learnable prompts for authentic and manipulated classes,

Read the complete text