TsallisPGD: Adaptive Gradient Weighting for Adversarial Attacks on Semantic Segmentation

Alexander Matyasko, Xin Lou, Indriyati Atmosukarto, Wei Zhang · 2026

Read TsallisPGD: Adaptive Gradient Weighting for Adversarial Attacks on Semantic Segmentation by Alexander Matyasko, Xin Lou, Indriyati Atmosukarto, Wei Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03405.txt.

Introduction

Alexander Matyasko, Xin Lou, Indriyati Atmosukarto, and Wei Zhang introduce TsallisPGD, an adversarial attack on semantic segmentation models. Flipping thousands of pixel labels at once is far harder than fooling a classifier, and standard cross-entropy wastes effort on pixels already fooled, overstating robustness.

Their attack uses Tsallis cross-entropy, whose parameter steers gradient focus across pixel confidence levels. No single fixed value wins everywhere, so a dynamic schedule sweeps it during optimization. On three datasets, one validation-chosen schedule gave the best average rank against five rival attacks, on both standard and robust models.

Catalog record

CREATORAlexander Matyasko, Xin Lou, Indriyati Atmosukarto, Wei Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03405

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TsallisPGD: Adaptive Gradient Weighting for Adversarial Attacks on Semantic Segmentation Thanks: Accepted to the 2026 IJCNN (author’s accepted version). © 2026 IEEE. Alexander Matyasko, Xin Lou, Indriyati Atmosukarto, Wei Zhang Abstract Attacking semantic segmentation models is significantly harder than image classification models because an attacker must flip thousands of pixel predictions simultaneously. Standard pixel-wise cross-entropy (CE) is ill-suited to this setting: it tends to overemphasize already-misclassified pixels, which slows optimization and overstates model robustness. To address these issues, we introduce TsallisPGD, an adversarial attack built on the Tsallis cross-entropy, a generalization of CE parameterized by q q , which adaptively reshapes the gradient landscape by controlling gradient concentration across pixels. By varying q q , we steer the attack toward pixels at different confidence levels. We first show that no single fixed- q q is universally optimal, as its effectiveness depends on the dataset, model architecture, and perturbation budget. Motivated by this, we propose a dynamic q q -schedule that sweeps q q during optimization. Extensive experiments on Cityscapes, Pascal VOC, and ADE20K show that TsallisPGD, using a single validation-selected schedule, achieves the best average attack rank across all evaluated settings and improves over CEPGD, SegPGD, CosPGD, JSPGD, and MaskedPGD in reducing accuracy and mIoU on both standard and robust models. I Introduction Deep neural networks (DNNs) have proven remarkably effective at tackling complex computer vision tasks, such as image classification [ 26 , 14 ] , object detection [ 24 , 19 ] , semantic segmentation [ 5 , 33 ] , depth estimation [ 11 , 4 ] , and optical flow estimation [ 22 , 29 ] . These advances have significant applications in autonomous driving [ 35 ] and biomedical image analysis [ 18 ] . However, despite these impressive advancements, deep neural networks face critical challenges that limit their broader adoption in safety- and security-critical applications, such as self-driving cars and medical imaging. Specifically, DNNs are sensitive to distribution shifts such as changes in lighting and weather conditions [ 28 , 23 ] . Most critically, state-of-the-art DNNs are vulnerable to adversarial attacks, which can craft small, imperceptible input perturbations that cause severe and unexpected model failures [ 27 , 12 ] . Since the discovery of adversarial attacks against image classification models, significant progress has been made in evaluating and improving the robustness of image classification models [ 7 , 21 ] . Recent research has broadened to consider a wider range of prediction tasks [ 32 , 30 ] and input modalities [ 15 , 17 ] . Semantic segmentation presents unique challenges for adversarial robustness compared to image classification. Although both types of attacks aim to optimize a loss function that measures attack success, adversarial attacks on semantic segmentation are significantly more challenging because they must alter predictions for all pixels simultaneously. Likewise, improving the robustness of semantic segmentation models is more difficult, as any adversarial defense must ensure that predictions for all pixels remain robust to all feasible perturbations. Early approaches for attacking semantic segmentation models [ 32 , 2 ] directly adapted the projected gradient descent (PGD) attack to the semantic segmentation setting. These attacks maximize the average per-pixel loss, e.g. cross-entropy, across all output pixels, mirroring the training objective. While maximizing average per-pixel loss is a straightforward extension of PGD, it ignores the multi-output nature of segmentation models, leading to suboptimal optimization dynamics. As a result, PGD-based attacks on semantic segmentation models often suffer from slow convergence, requiring hundreds of attack iterations and potentially leading to an overly optimistic assessment of model robustness. To address these limitations, recent works have focused on adaptively reweighting per-pixel losses to improve optimization dynamics. SegPGD [ 13 ] assigns different weights to correctly and incorrectly classified pixels. SegPGD dynamically anneals pixel weights, initially focusing exclusively on correctly classified pixels and then gradually shifting attention to a mixture of correctly and incorrectly classified pixels. CosPGD [ 1 ] sets pixel weights proportional to the cosine similarity between the original and adversarial outputs, effectively prioritizing pixels where the model is more confident. Croce et al. [ 8 ] further observed that gradients of standard cross-entropy saturate on misclassified pixels, causing attacks to overemphasize pixels that are already successfully attacked. To mitigate this issue, they proposed alternative objectives with bounded gradients on misclassified pixels, such as the Jensen–Shannon divergence and masked cross-entropy, which yield more stable optimization dynamics. In this paper, we propose TsallisPGD , an adversarial attack for semantic segmentation built on Tsallis cross-entropy, a generalization of standard cross-entropy controlled by a parameter q . A key property of Tsallis cross-entropy is that it induces a confidence-dependent reweighting of pixels in gradient space: each pixel is effectively scaled by p_{y}^{1-q} , where p_{y} is the predicted probability of the ground-truth class. For q<1 , this reshapes optimization by emphasizing high-confidence, correctly classified pixels while down-weighting high-confidence, incorrectly classified ones, making it well suited for segmentation attacks. We analyze the resulting gradient behavior and show that q controls both the magnitude and the location of the gradient peak. By adjusting q , the attack can thus target different subsets of pixels. We further show empirically that no single fixed- q is optimal across datasets, architectures, and perturbation budgets. This motivates a dynamic q -schedule that sweeps q during optimization enabling the attack to target pixels at different confidence levels. We select the best q -schedule using a held-out validation split and apply the same selected schedule across the entire test benchmark with no further tuning. Extensive experiments on Cityscapes , Pascal-Voc , and Ade20K show that TsallisPGD achieves the best overall average rank across both standard and adversarially trained models. Our attack improves over CEPGD, SegPGD, CosPGD, JSPGD, and MaskedPGD in reducing both accuracy and mIoU on aggregate. Code will be available at https://github.com/aam-at/tsallis_pgd . To summarize, our main contributions are as follows: • We introduce TsallisPGD, a novel adversarial attack for semantic segmentation that leverages Tsallis cross-entropy to control the attack focus via parameter q . • We provide a theoretical analysis showing that Tsallis cross-entropy induces a confidence-dependent gradient reweighting p_{y}^{1-q} , and demonstrate how q controls both the location and magnitude of a lower-bound proxy for the gradient peak. • We propose a dynamic q -schedule that varies q over the course of optimization, enabling the attack to target pixels at different confidence levels. • We demonstrate on Cityscapes , Pascal-Voc , and Ade20K that TsallisPGD achieves the best overall average rank among strong baselines on both standard and adversarially trained models. II Related Work Adversarial attacks against classification models: Seminal work by Szegedy et al. [ 27 ] revealed that state-of-the-art image classifiers are highly vulnerable to adversarial perturbations—carefully crafted, small and often imperceptible input changes that can fool the model. Goodfellow et al. [ 12 ] introduced the Fast Gradient Sign Method (FGSM), a computationally efficient one-step attack that perturbs inputs in the direction of the gradient of the loss with respect to the input. Kurakin et al. [ 16 ] and Madry et al. [ 21 ] introduced stronger iterative attacks based on FGSM, such as BIM (I-FGSM) and PGD, respectively. Madry et al. [ 21 ] further formalized PGD

Read the complete text