Benchmarking Adversarial Patch Selection and Location
Shai Kimhi, Avi Mendelson, Moshe Kimhi · 2025
Read Benchmarking Adversarial Patch Selection and Location by Shai Kimhi, Avi Mendelson, Moshe Kimhi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01676.txt.
Introduction
Shai Kimhi, Avi Mendelson, and Moshe Kimhi map in 2025 where small adversarial patches hurt vision models most. A patch covering two percent of an image can flip a confident label. Where the patch sits may matter as much as what it shows.
Their benchmark, PatchMap, runs over 150 million forward passes across ImageNet validation images. Hot spots emerge where patches reliably tank confidence. A simple trick uses segmentation masks to place patches on weak regions, with no gradient queries at all. Attack success rises eight to thirteen points over random or fixed spots across five model types. The full map is slated for public release to spur location-aware defenses.
Catalog record
| CREATOR | Shai Kimhi, Avi Mendelson, Moshe Kimhi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01676
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking Adversarial Patch Selection and Location Shai Kimhi, Avi Mendelson, Moshe Kimhi Abstract Adversarial patch attacks threaten the reliability of modern vision models. We present PatchMap , the first spatially exhaustive benchmark of patch placement, built by evaluating over 1.5 × 10 8 1.5\times 10^{8} forward passes on ImageNet validation images. PatchMap reveals systematic “hot-spots” where small patches (as little as 2% of the image) induce confident misclassifications and large drops in model confidence. To demonstrate its utility, we propose a simple segmentation-guided placement heuristic that leverages off-the-shelf masks to identify vulnerable regions without any gradient queries. Across five architectures-including adversarially trained ResNet-50-our method boosts attack success rates by 8–13 percentage points compared to random or fixed placements. We publicly release PatchMap 1 1 1 huggingface.co/datasets/PatchMap/PatchMap_v1 and the code implementation 2 2 2 github.com/PathMap7/PatchMap . The full PatchMap bench (6.5 B predictions, multiple backbones) will be release upon acceptance to further accelerate research on location-aware defenses and adaptive attacks. Figure 1 : PatchMap evaluation method: We apply patch for any location in the image, and for each location we estimate the predicted class of the model and the confidence of the model on the ground truth class ( y ). tacks. 1 Introduction Deep neural networks have achieved near–human accuracy on many vision benchmarks, yet they remain alarmingly brittle to small, localized perturbations. In particular, adversarial patches -printed stickers or posters that can be physically applied to a scene-have been shown to reliably fool classifiers and detectors in both digital and real‐world settings [ 1 , 3 , 18 ] . Their practical threat spans self‐driving cars, surveillance systems, and biometric authentication. Despite extensive work on crafting patch appearance -optimizing texture, shape, and color-little attention has been paid to the equally critical question of where to place a patch. Most defenses and evaluation protocols implicitly assume a fixed placement (e.g. center or corner of the image) or explore joint location‐texture optimization on a handful of examples [ 15 , 21 ] . Without a systematic, large‐scale study of spatial vulnerability, researchers cannot reliably benchmark location‐aware attacks, nor can they design defenses that guard against worst‐case placements. Motivation. Consider two identical patches: one pasted over a salient object region, the other over background. Intuitively, the former will more strongly disrupt classification, yet existing benchmarks provide only sparse evidence for this intuition. A comprehensive spatial analysis would answer: Which regions of a natural image are most susceptible? How does vulnerability vary across object classes, patch size, and model architecture? And crucially, can we exploit this knowledge to mount faster, more effective attacks without model queries? PatchMap. We introduce PatchMap , the first spatially exhaustive benchmark for adversarial patch placement. Starting from the ten transferable patches of the ImageNet‐Patch dataset [ 12 ] , we paste each patch at every stride‐2 location and three scale factors (50, 25, 10 px) on all 50k ImageNet‐1K validation images. This yields over 1.0\times 10^{8} forward passes on a standard ResNet‐50, recording both predicted labels and softmax confidences at each location. These dense “vulnerability maps” uncover stable hot‐spots, quantify confidence collapse patterns, and reveal spatial motifs invisible to prior small‐scale studies. Contributions 1. PatchMap dataset. A public release of 100 M+ location‐conditioned predictions on HuggingFace , scaling to 6.5 B entries. 2. Rigorous analysis. Unified definitions of attack‐success rate (ASR) and confidence drop ( \Delta\!\mathrm{conf} ), evaluated across location, patch size, and model robustness, exposing recurring spatial vulnerabilities. 3. Segmentation‐guided placement. A fast, zero‐gradient heuristic that selects high‐impact locations via off‐the‐shelf semantic masks, boosting ASR by 8–13 pp over random or fixed baselines-even on adversarially trained networks. By decoupling what a patch looks like from where it lands, PatchMap lays the groundwork for location‐aware defenses, adaptive attacks, and deeper insight into the spatial dynamics of adversarial vulnerability. 2 Related Work 2.1 Universal and Physical Adversarial Patches [ 1 ] first introduced universal patches that consistently coerce a classifier into an attacker–chosen label. Because these patches are physically realizable , later studies printed them to fool real-world systems: LaVAN places a small visible blob on the background [ 6 ] , while RP2 stickers make stop-signs read as speed-limits [ 3 ] . Domain-specific variants exist for face recognition, where perturbations are embedded in eyeglass frames [ 18 ] , and for traffic-sign detection [ 20 ] . The feasibility of such localised, robust attacks motivated datasets such as ImageNet-Patch [ 12 ] , which supplies transferable patch textures later reused in our work. 2.2 Patch Placement and Location Optimisation Early attacks fixed the patch at a predetermined corner or the object centre [ 6 , 3 ] , implicitly assuming position mattered little. Subsequent research relaxed this by optimising location jointly with texture: LOAP employs gradient descent on both variables [ 15 ] ; Simultaneous-Patch uses reinforcement learning (RL) to search position in a black-box setting [ 21 ] ; PatchAttack learns an RL policy subject to a query budget [ 24 ] . Real-world “Adversarial Sticker” work further optimises in-plane rotation and scale of printed patches [ 22 ] . Generative Dynamic Patch Attack (GDPA) [ 11 ] a use GANs to propose both content and location, showing that placement is tightly coupled with texture generation. Attention-guided methods such as PS-GAN [ 13 ] infer high-impact regions from saliency maps, while Shapeshifter extends the idea to object detection [ 3 ] . Despite this progress, prior studies either optimise on a handful of images or evaluate sparse locations; none offers an exhaustive , publicly released map of spatial vulnerability. 2.3 Context Awareness and Adaptive Attacks Modern attacks adapt the patch to scene context. Distributed stickers cover multiple object parts for occlusion robustness [ 22 ] , and Dynamic-Patch sequences adjust appearance over time [ 11 ] . Beyond classification, context-aware patches fool detectors, depth estimators, and segmentation networks, often by incorporating physical transforms (viewpoint, lighting) into training [ 3 ] . Several works exploit semantic priors: masking patches to background only, or using segmentation cues to stay off salient regions [ 14 ] . Our segmentation-guided heuristic takes the opposite tack- seeking the most segmentation-confident region-while remaining optimisation-free and substantially faster than RL or gradient-based search. PatchMap complements the above literature by offering the first large‐scale, architecture-agnostic benchmark that decouples patch appearance from exhaustive placement, enabling quantitative comparison of any optimisation-free or optimisation-based strategy on common ground. 3 Dataset Design Patch source. We use the ten publicly released ImageNet-Patch adversaries of Li et al. 2022 . Each 50\times 50 RGB patch is gradient–optimised for a distinct target class and shown to transfer across common CNN backbones. Fixing this set guarantees strict reproducibility and avoids the confound of re-optimising patch content. The patches are applied without rotation. Spatial sweep. For every ImageNet-1K validation image ( 224\times 224 ), each patch is pasted on a dense stride-2 grid of feasible centres. The grid contains 112\times 112=12{,}544 positions and is evaluated at three square sizes: native 50\!\times\!50 , down-scaled 25\!\times\!25 , and 10\!\times\!10 px covering almost the whole image while keeping compute tractable. In total 50,000 images \times 10 patches \times 3 sizes \times 12,544 locations \approx 1.9\times 10^{10} potential placements. By batching inference and discarding placements that would