Auto-ART : Structured Literature Synthesis and Automated Adversarial Robustness Testing across Evaluation Protocols, Multi-Attack Threat Models, and LLM-Driven Breaking

Abhijit Talluri · 2026

Read Auto-ART : Structured Literature Synthesis and Automated Adversarial Robustness Testing across Evaluation Protocols, Multi-Attack Threat Models, and LLM-Driven Breaking by Abhijit Talluri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20704.txt.

Introduction

Talluri makes two contributions to adversarial robustness testing. First, synthesis: he studies nine corpus sources from 2020 to 2026 through seven rules, giving the first analysis of the field's consensus and gaps. Second, Auto-ART: an open framework with over 50 attacks, 28 defense modules, the RDI, and gradient-masking detection. Tests on RobustBench show Auto-ART's pre-screening spots gradient masking in 92 percent of cases, and RDI rankings match full AutoAttack. Multi-norm tests expose a 23.5 point gap between average and worst-case robustness.

Catalog record

CREATORAbhijit Talluri
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75109
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20704

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Auto-ART : Structured Literature Synthesis and Automated Adversarial Robustness Testing across Evaluation Protocols, Multi-Attack Threat Models, and LLM-Driven Breaking Abhijit Talluri Abstract Adversarial robustness evaluation underpins every claim of trustworthy ML deployment, yet the field suffers from fragmented protocols, undetected gradient masking, single-norm tunnel vision, and premature generalization from stylized benchmarks to production code. We make two contributions. (1) Structured synthesis. We analyze nine peer-reviewed corpus sources (ICML, NeurIPS, ICLR, UAI; 2020–2026) through seven complementary protocols—citation-chain tracing, gap scanning, methodology auditing, cross-paper synthesis, assumption stress-testing, knowledge mapping, and lay-accessible distillation—producing the first end-to-end structured analysis of what the adversarial evaluation community agrees on, disagrees about, and cannot yet resolve . (2) Auto-ART framework. We introduce Auto-ART [ 1 ] , an open-source framework that directly operationalizes every identified gap: 50+ attacks across seven categories, 28 defence modules, the Robustness Diagnostic Index (RDI; ∼ ⁣ × {\sim}30\!\times faster screening), First-Order Stationarity Condition (FOSC) gradient-masking detection, multi-norm evaluation ( ℓ 1 / ℓ 2 / ℓ ∞ \ell_{1}/\ell_{2}/\ell_{\infty} /semantic/spatial) with worst-case reporting, adaptive memory-guided attack selection, and compliance mapping to NIST AI RMF, OWASP LLM Top 10, and the EU AI Act. Empirical validation on the CIFAR-10 RobustBench leaderboard demonstrates that Auto-ART ’s pre-screening gate correctly identifies gradient masking in 92% of flagged configurations, that RDI rankings agree with full AutoAttack within Kendall τ ≥ 0.81 \tau\!\geq\!0.81 , and that multi-norm evaluation exposes a 23.5 pp gap between average and worst-case robustness on state-of-the-art models. Three novel technical directions—worst-case multi-norm adversarial training curricula, architecture-conditional RDI calibration for Vision Transformers and LLMs, and ecologically valid agent red-teaming benchmarks—are proposed with concrete feasibility assessments and ablation designs. To the best of our knowledge, no prior work combines structured meta-scientific analysis with an executable evaluation framework bridging literature gaps directly into engineering. Keywords: adversarial robustness, evaluation protocols, gradient masking, multi-attack benchmarks, diffusion purification, LLM red teaming, automated adversarial testing, NIST AI RMF, OWASP. 1 Introduction Machine-learning models deployed in safety-critical perception [ 2 , 3 ] , language understanding [ 4 ] , and decision-making [ 5 ] are expected to resist adversarially chosen input perturbations. A decade of work on adversarial training [ 3 , 6 ] and certified defences [ 7 ] has produced steady progress on standard benchmarks. Yet a recurring pattern undermines confidence in these numbers: defences are proposed with optimistic robustness claims that collapse under stronger attacks, richer threat models, or correct gradient accounting [ 8 , 9 , 10 , 11 ] . The cycle repeats because the failures are not merely technical but structural—rooted in evaluation practice itself: (i) Reliance on single \ell_{p} norms when real adversaries combine spatial, semantic, and compression perturbations [ 5 ] ; (ii) Gradient masking inflating attack-based estimates across both deterministic [ 11 ] and stochastic [ 10 ] pipelines; (iii) Under-sampling of randomized-defence draws [ 10 ] ; (iv) Premature transfer from stylized CTF tasks to production code [ 4 ] . We address these failures through two complementary contributions. Contribution 1: Structured literature synthesis. We analyse a curated corpus of nine verified sources (Table 1 ) using seven protocols drawn from systematic review methodology. Unlike narrative surveys that summarise papers sequentially, our synthesis traces intellectual lineages (who originated, challenged, and refined each concept), identifies testable assumptions the literature relies on but never validates, and maps the knowledge topology of the field. To the best of our knowledge, this is the first application of structured meta-scientific analysis—combining citation-chain tracing, gap scanning, methodology auditing, assumption stress-testing, and knowledge mapping in a single manuscript—to adversarial ML evaluation. Contribution 2: Auto-ART framework. We introduce Auto-ART ( Auto mated A dversarial R obustness T esting) [ 1 ] , an open-source Python framework that directly operationalises every gap identified in the synthesis (Section 11 ). Key capabilities: • 50+ attacks across evasion (37+), poisoning (10+), extraction (3), inference (8+), audio (2), NLP/text, LLM, and agentic categories; • 28 defence modules including TRADES [ 6 ] , AWP, OAAT, certified training, guardrails, and circuit breakers; • Pre-screening gate: RDI screening ( {\sim}30\!\times faster) + FOSC gradient-masking detection before expensive attack runs; • Multi-norm evaluation across \ell_{1}/\ell_{2}/\ell_{\infty} /semantic/spatial with worst-case reporting; • Adaptive attack selection with memory-guided tiered escalation; • Compliance mapping to NIST AI RMF, OWASP LLM Top 10, and the EU AI Act; • CI/CD integration via SARIF 2.1.0 output with 346+ automated tests. Paper organisation. Section 2 surveys related work. Section 3 describes corpus selection, the intake protocol, and the contradiction analysis. Sections 4 – 10 present the seven structured analyses. Section 11 describes the Auto-ART architecture. Section 12 presents empirical evaluation. Section 13 proposes three novel technical directions with feasibility assessments. Section 14 discusses implications, and Section 15 concludes. 2 Related Work Adversarial robustness foundations. Goodfellow et al. [ 2 ] introduced FGSM, demonstrating that linear perturbations suffice to fool DNNs. Carlini and Wagner [ 12 ] showed that defensive distillation [ 13 ] fails against optimisation-based attacks, establishing the arms race paradigm. Madry et al. [ 3 ] formalised PGD adversarial training as a min-max optimisation, defining the standard \ell_{\infty} robustness benchmark on CIFAR-10 at \varepsilon\!=\!8/255 . Athalye et al. [ 9 ] catalogued obfuscated gradients as a systematic failure mode, breaking seven ICLR 2018 defences. Evaluation standardisation. Croce and Hein [ 8 ] assembled AutoAttack—a parameter-free ensemble of APGD-CE, APGD-DLR, FAB, and Square Attack—re-evaluating > 50 defended models and documenting > 10 pp overestimation in most cases. RobustBench [ 14 ] operationalised this into a living leaderboard. Zhang et al. [ 6 ] proposed TRADES, decomposing robust loss into clean and adversarial terms; Li et al. [ 11 ] later showed that TRADES itself suffers gradient masking under certain hyperparameters. Multi-attack and continual robustness. Dai et al. [ 5 ] introduced MultiRobustBench (16 models \times 9 attacks \times 20 strengths), revealing worst-case multi-attack robustness below random guessing. Dai et al. [ 15 ] addressed temporal attack evolution through continual robust training with logit-space regularisation. Stochastic defence evaluation. Cohen et al. [ 7 ] established randomised smoothing for certified \ell_{2} robustness. Kassis et al. [ 10 ] demonstrated that diffusion-based purification (DBP) collapses under correct gradient computation and proper randomness sampling, breaking even majority-vote variants. Xiao et al. [ 16 ] independently confirmed these findings through DiffHammer, a systematic framework for evaluating DBP robustness across multiple purification strategies. Beyond stochastic defences, Guo et al. [ 17 ] identified imbalanced gradients —a gradient masking phenomenon distinct from obfuscated gradients, where one loss term’s gradient dominates, pushing attacks suboptimally—and proposed the Margin Decomposition (MD) attack that exceeds AutoAttack on 16/20 defence models. He et al. [ 18 ] introduced SoundnessBench, revealing bugs in three established neural network verifiers and underscoring that certification tools themselves require rigorous validation. Automated adversarial testing. The IBM Adversarial Robustness Toolbox (ART) [ 19 ] provides attack/defence primitives. Microsoft Counterfit [ 20 ] targets ML model security assessments. Garak [ 21 ] scans LLM vulnerabilities across {\sim}100 attack vectors. PyRIT [ 22 ] automates multi-turn red teaming. HarmBench [ 23 ] standardises LLM red teaming across 18 attack methods and 33 target models. NIST AI 100-2e2025 [ 24 ] provides the authoritative taxonomy of adversarial attacks and mitigations. Carlini et al.

Read the complete text