Context-Aware Weakly Supervised Image Manipulation Localization with SAM Refinement
Xinghao Wang, Tao Gong, Qi Chu, Bin Liu, Nenghai Yu · 2025
Read Context-Aware Weakly Supervised Image Manipulation Localization with SAM Refinement by Xinghao Wang, Tao Gong, Qi Chu, Bin Liu, Nenghai Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20294.txt.
Introduction
Wang, Gong, Chu, Liu, and Yu locate faked regions in images while training with only image-level labels, in a paper from twenty twenty-five. Pixel-level labels for fakes are costly to make by hand. Weak labels say only whether an image is faked at all. Past weak methods ignore edge cues, so their maps are blurry and imprecise.
Their boundary module gathers edge features and learns context mismatch near edges. A second module refines maps by guiding a large segmentation model with activation heat maps. The two parts sit in a dual-branch network that mixes transformers and convolutions. Tests on several data sets show outstanding accuracy for a weakly trained method. The maps hug true fake boundaries far better than before.
Catalog record
| CREATOR | Xinghao Wang, Tao Gong, Qi Chu, Bin Liu, Nenghai Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20294
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Context-Aware Weakly Supervised Image Manipulation Localization with SAM Refinement Xinghao Wang, Tao Gong, Qi Chu, Bin Liu, Nenghai Yu Abstract Malicious image manipulation poses societal risks, increasing the importance of effective image manipulation detection methods. Recent approaches in image manipulation detection have largely been driven by fully supervised approaches, which require labor-intensive pixel-level annotations. Thus, it is essential to explore weakly supervised image manipulation localization methods that only require image-level binary labels for training. However, existing weakly supervised image manipulation methods overlook the importance of edge information for accurate localization, leading to suboptimal localization performance. To address this, we propose a Context-Aware Boundary Localization (CABL) module to aggregate boundary features and learn context-inconsistency for localizing manipulated areas. Furthermore, by leveraging Class Activation Mapping (CAM) and Segment Anything Model (SAM), we introduce the CAM-Guided SAM Refinement (CGSR) module to generate more accurate manipulation localization maps. By integrating two modules, we present a novel weakly supervised framework based on a dual-branch Transformer-CNN architecture. Our method achieves outstanding localization performance across multiple datasets. I Introduction With the proliferation of maliciously forged images, the localization of manipulated regions has become critical for combating misinformation. Currently, most image manipulation localization methods leverage both image-level binary classification labels and pixel-level mask annotations for fully supervised training [ 1 , 2 ] . Early approaches leveraged noise, boundaries, and color inconsistencies [ 3 , 4 ] , while recent methods focus on content-agnostic features: CAT-Net [ 5 ] learns compression artifacts; MVSS-Net [ 6 ] uses multi-scale noise; Mantra-Net [ 7 ] detects operational boundaries; and PROMPT-IML [ 8 ] aligns semantic and high-frequency features via pre-trained models. Fig. 1: The framework of our method: We integrate our custom-designed CABL module and CGSR module into the backbone. We train the model using only images and image-level binary labels. Sobel [ 9 ] denotes the Sobel operator, Stem generates patch embeddings, ConvBlock and TransBlock [ 10 ] are composed of convolution layers and transformer [ 11 ] layers respectively. The green point in the Prompt Generator indicates the positive point prompt, the red point represents the negative point prompt, and the red box denotes the bounding box prompt. Fig. 2: Right of vertical line: first row (w/o CABL) shows feature maps of first 4 blocks; second row (w/ CABL) shows enhanced maps. Arrows indicate deeper block directions. Despite significant advances in fully supervised image manipulation localization methods [ 1 , 12 ] , the growing diversity and complexity of manipulation techniques have made acquiring comprehensive pixel-level mask annotations increasingly impractical. Dense annotation of manipulated regions is both challenging and time-consuming. As a result, recent research has shifted towards weakly supervised approaches to enhance generalization and localization performance. For instance, WSCL [ 13 ] introduces multi-source consistency and inter-patch consistency modules by exploiting the self-consistency properties of manipulated images. While effective for manipulation detection tasks, its localization performance is limited by inadequate consideration of boundary information, which is crucial for accurate manipulation localization. Image manipulation techniques like copy-move [ 14 , 15 ] and splicing [ 16 ] involve non-generative operations in which manipulated regions are copied from authentic areas. As a result, the noise and texture within manipulated regions often resemble those of real regions, making the boundaries of manipulated areas crucial for localization. To better learn boundary information, we propose a Context-Aware Boundary Localization (CABL) module that utilizes context-inconsistency at the boundaries to more precisely identify manipulated regions. Furthermore, previous methods [ 13 , 17 ] in weakly supervised image manipulation localization have largely overlooked the potential of Class Activation Mapping (CAM) [ 18 ] . In our approach, we compute CAM maps using the output of a dual-branch Transformer-CNN network to generate coarse masks of manipulated regions. To refine these coarse masks, we leverage the strong capabilities of the Segment Anything Model (SAM) [ 19 ] through a CAM-Guided SAM Refinement (CGSR) module. Specifically, we use the bounding rectangle of the largest connected component from the coarse masks and the positive and negative sampling points from the CAM [ 18 ] without maximum-minimum normalization as visual prompts for SAM [ 19 ] . SAM [ 19 ] can generate more accurate localization masks using these visual prompts, without requiring additional training. By integrating these two modules, we propose a novel framework that combines CABL and CGSR within a dual-branch Transformer-CNN architecture. This framework can learn the context-inconsistency of the manipulated images under weakly supervised training conditions, enabling precise localization of the manipulated regions. Experiments validate that our method achieves state-of-the-art performance in both manipulation detection and localization, effectively identifying and segmenting manipulated regions across diverse scenarios. The principal contributions of this work include: (1) We propose the Context-Aware Boundary Localization (CABL) module that explicitly learns context-inconsistency at manipulated boundaries, enabling precise localization of manipulated regions. (2) We propose the CAM-Guided SAM Refinement (CGSR) module , employing CAM [ 18 ] to generate the coarse-grained masks and achieving more precise localization results base on SAM’s [ 19 ] powerful segmentation capabilities without additional training. (3) We train the model using only image-level binary classification labels from the CASIAv2 [ 20 ] dataset and evaluate its performance on four unseen test datasets, demonstrating its effectiveness. II METHODOLOGY II-A Overview TABLE I: Comparison of Image Manipulation Detection and Localization Methods across Different Datasets 1,2 Method CASIAv1 Columbia Coverage IMD2020 Average I-AUC P-F1 I-AUC P-F1 I-AUC P-F1 I-AUC P-F1 I-AUC P-F1 NOII [ 21 ] 0.500 0.157 0.500 0.311 0.500 0.205 0.500 0.124 0.500 0.199 Un. 3 CFAl [ 22 ] 0.482 0.140 0.344 0.320 0.525 0.188 0.500 0.111 0.500 0.190 Mantra-Net [ 23 ] 0.141 0.155 0.701 0.364 0.490 0.286 0.719 0.122 0.513 0.231 CAT-Net [ 5 ] 0.630 0.276 0.782 0.352 0.572 0.134 0.721 0.102 0.693 0.216 MVSS-Net [ 6 ] 0.937 0.452 0.980 0.638 0.731 0.453 0.656 0.260 0.826 0.451 GSR-Net [ 24 ] 0.502 0.387 0.502 0.613 0.515 0.285 0.505 0.175 0.506 0.365 Full. FCN+DA [ 1 ] 0.796 0.441 0.762 0.223 0.541 0.199 0.746 0.270 0.711 0.283 MIL-FCN [ 25 ] 0.647 0.117 0.807 0.089 0.542 0.121 0.578 0.097 0.644 0.106 MIL-FCN [ 25 ] +WSCL [ 13 ] 0.829 0.172 0.920 0.270 0.584 0.178 0.733 0.193 0.766 0.203 Araslanov and Roth [ 26 ] 0.642 0.112 0.773 0.102 0.560 0.127 0.665 0.094 0.660 0.109 Araslanov and Roth [ 26 ] +WSCL [ 13 ] 0.796 0.153 0.917 0.362 0.591 0.201 0.701 0.173 0.751 0.222 Weak. Ours 0.816 0.300 0.963 0.382 0.658 0.190 0.783 0.368 0.805 0.310 1 The best and second best results in the weakly supervised method are indicated in boldface and underlined , respectively. 2 All localization results in our method were refined using CGSR. 3 Un. means unsupervised methods; Full. means fully supervised methods; Weak. means weakly supervised methods. TABLE II: Ablation Study of CABL and CGSR Modules on Coverage and IMD2020 Datasets CABL CGSR Coverage IMD2020 Avg I-AUC P-F1 I-AUC P-F1 I-AUC P-F1 - - 0.649 0.154 0.767 0.342 0.708 0.248 ✓ - 0.658 0.177 0.783 0.361 0.721 0.269 - ✓ 0.649 0.175 0.767 0.353 0.708 0.264 ✓ ✓ 0.658 0.190 0.783 0.368 0.721 0.279 TABLE III: Ablation Study of Different CABL Structures on Coverage and IMD2020 Datasets 1 Structure