Adversarially-Aware Architecture Design for Robust Medical AI Systems
Alyssa Gerhart 1 (Student), Balaji Iyangar 2 (Associate Professor) · 2025
Read Adversarially-Aware Architecture Design for Robust Medical AI Systems by Alyssa Gerhart 1 (Student), Balaji Iyangar 2 (Associate Professor) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23622.txt.
Introduction
Alyssa Gerhart and Balaji Iyangar examine how easily medical AI can be fooled by adversarial attacks, tiny crafted changes to an input that humans barely notice. In healthcare, a fooled model can misclassify a case, delay treatment, or mislead a diagnosis, with the worst risks falling on underserved patients. Design choices, not just training data, shape how exposed a system is.
Working with a dermatology dataset, the authors map the threats, run attack benchmarks, and measure how far accuracy falls. They then test defenses such as adversarial training and distillation, which cut attack success but must be weighed against accuracy on clean images. Their results show partial protection at best from any single defense. They close by urging combined technical, ethical, and policy measures to build medical AI that is both resilient and fair.
Catalog record
| CREATOR | Alyssa Gerhart 1 (Student), Balaji Iyangar 2 (Associate Professor) |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23622
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adversarially-Aware Architecture Design for Robust Medical AI Systems Alyssa Gerhart 1 (Student), Balaji Iyangar 2 (Associate Professor) Abstract Adversarial attacks pose a severe risk to AI systems used in healthcare, capable of misleading models into dangerous misclassifications that can delay treatments or cause misdiagnoses. These attacks, often imperceptible to human perception, threaten patient safety, particularly in underserved populations. Our study explores these vulnerabilities through empirical experimentation on a dermatological dataset, where adversarial methods significantly reduce classification accuracy. Through detailed threat modeling, experimental benchmarking, and model evaluation, we demonstrate both the severity of the threat and the partial success of defenses like adversarial training and distillation. Our results show that while defenses reduce attack success rates, they must be balanced against model performance on clean data. We conclude with a call for integrated approaches—technical, ethical, and policy-based—to build more resilient, equitable AI in healthcare. I Introduction Artificial Intelligence (AI) has revolutionized various sectors of society, and healthcare stands as one of the most impacted. Deep learning models, trained on large and diverse medical datasets, are increasingly being used to aid clinical diagnosis, treatment planning, and patient monitoring. In fields such as dermatology, AI-powered systems have demonstrated capabilities comparable to, or in some cases even exceeding, human expert performance. However, as the reliance on these systems grows, so too does the importance of understanding their vulnerabilities. One particularly concerning challenge is the susceptibility of these models to adversarial attacks—strategically crafted input modifications that deceive the AI system into making incorrect predictions, often with high confidence [ 1 ] . These perturbations can be minute, invisible to the human eye, and yet they can cause a malignant lesion to be misclassified as benign, potentially delaying life-saving treatment. The implications of such vulnerabilities are profound. Not only do they erode trust in medical AI among clinicians and patients, but they also exacerbate disparities in healthcare access and outcomes—particularly in underrepresented or underserved communities where AI is often proposed as a stopgap for the shortage of medical professionals. Moreover, AI systems deployed in real-world settings may encounter adversarial threats not just from malicious actors but also from unintentionally flawed data or systemic bias. Therefore, it is crucial to design and evaluate models that are not only accurate but also robust and ethically reliable. In this paper, we contribute to the growing body of research on adversarial robustness in medical AI by addressing three central objectives. First, we characterize the types of adversarial threats relevant to healthcare settings, emphasizing their mechanisms and potential impact. Second, we conduct extensive empirical evaluations on the publicly available dermatological data set for skin cancer, namely ISIC [ 2 ] , using Nightshade, a tool that turns any image into a data sample that is unsuitable for model training. More precisely, Nightshade transforms images into ”poison” samples, so that models training on them without consent will see their models learn unpredictable behaviors that deviate from expected norms, e.g. a prompt that asks for an image of a cow flying in space might instead get an image of a handbag floating in space [ 3 ] . Third, we assess defense techniques including adversarial training, defensive distillation, and hybrid methods, evaluating their trade-offs between robustness and model accuracy on clean, unperturbed data. Our findings support the need for multifaceted approaches that combine technical defenses with ethical and policy-driven design principles. II Threat Model Analysis Adversarial threats targeting medical AI systems can be broadly categorized into three domains: data poisoning attacks, evasion attacks, and model extraction or reverse engineering attacks. Each poses a unique challenge to model reliability, patient safety, and data security. II-A Data Poisoning Attacks Data poisoning attacks compromise the integrity of a model during its training phase by embedding maliciously crafted inputs into the training set. These poisoned inputs appear benign to human observers and to standard validation checks, yet they subtly alter the model’s internal representations, steering it toward incorrect behavior when presented with specific triggers [ 4 ] . A powerful example of this is the Nightshade technique, which modifies image textures in a way that is imperceptible to clinicians but nonetheless introduces predictable classification errors. In dermatological applications, even a slight shift in skin texture, lesion border, or color pattern can mislead the model into mislabeling a malignant case as benign. This stealthy form of attack is especially dangerous because it does not require access to the deployed system—only to the training data pipeline. Once the poisoned data is included, the model unknowingly learns the attacker’s intended misclassification behavior, leading to long-term vulnerabilities that are difficult to detect and remediate post-deployment. (a) Original (b) Poisoned (c) Anchor Poison Attack Workflow Fig. 1: Illustration of a poisoning attack: (a) clean image, (b) poisoned variant, and (c) anchor causing the model to misclassify. II-B Evasion Attacks Unlike poisoning attacks, which corrupt a model during training, evasion attacks are executed at inference time by subtly modifying input samples after the model has already been trained. These attacks exploit the model’s learned decision boundaries by introducing imperceptible perturbations to the input, causing misclassification without visibly altering the image to the human eye. Prominent techniques such as the Fast Gradient Sign Method (FGSM) [ 7 ] and Projected Gradient Descent (PGD) [ 5 ] rely on manipulating the model’s gradients to craft these adversarial examples. In the context of medical imaging, the risks are amplified. For example, a chest X-ray or dermoscopic image with pixel-level perturbations may be misclassified as benign, even if pathological signs (e.g., a tumor or melanoma) are visually present. Studies have shown that deep learning models used for medical diagnosis are particularly vulnerable to adversarial examples, potentially leading to catastrophic clinical consequences [ 1 ] . The silent and undetectable nature of these perturbations, often imperceptible to clinicians, makes evasion attacks a serious threat in high-stakes, time-sensitive medical workflows [ 6 ] . II-C Model Extraction and Reverse Engineering A more insidious class of attacks involves model extraction, where adversaries probe a publicly accessible AI model—often through an API—to infer its internal structure and decision boundaries. By issuing carefully crafted queries and analyzing the corresponding outputs, attackers can construct a surrogate model that closely mimics the behavior of the target system. This technique was notably demonstrated by Tramèr et al. (2016), who showed that models deployed in the cloud can be effectively replicated using only black-box access [ 8 ] . The surrogate model can then be exploited to generate highly transferable adversarial examples, capable of deceiving both the surrogate and the original model, as discussed in studies on transferability of adversarial attacks (Papernot et al., 2017) [ 9 ] . More alarmingly, if the original model has been trained on sensitive medical or personal data, such extraction attacks may lead to unintended leakage of private information, a scenario explored by Carlini et al. (2021) in the context of membership inference and model inversion attacks [ 10 ] . These vulnerabilities raise serious concerns under data protection frameworks such as HIPAA and GDPR, underscoring the necessity for comprehensive safeguards. Such measures may include robust access controls, query rate-limiting, and model watermarking (Zhang et al., 2018) [ 11 ] , in addition to technical defenses against adversarial inputs. III