GF-Score: Certified Class-Conditional Robustness Evaluation with Fairness Guarantees

Arya Shah, Kaveri Visavadiya, Manisha Padala · 2026

Read GF-Score: Certified Class-Conditional Robustness Evaluation with Fairness Guarantees by Arya Shah, Kaveri Visavadiya, Manisha Padala (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12757.txt.

Introduction

Arya Shah and two colleagues audit certified robustness fairly across classes. Standard checks need costly attacks or give one aggregate score. GF-Score splits the GREAT Score into per-class profiles. Four welfare metrics measure disparity: disparity index, Gini coefficient, worst-case robustness, and fairness-penalized score. Self-calibration tunes temperature from clean accuracy alone. Tests on 22 RobustBench models across CIFAR-10 and ImageNet show exact splits. Per-class scores expose steady vulnerability patterns, with cat weakest in 76 percent of CIFAR-10 models. More robust models show greater class disparity.

Catalog record

CREATORArya Shah, Kaveri Visavadiya, Manisha Padala
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39020
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12757

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GF-Score: Certified Class-Conditional Robustness Evaluation with Fairness Guarantees Arya Shah, Kaveri Visavadiya, Manisha Padala Abstract Adversarial robustness is essential for deploying neural networks in safety-critical applications, yet standard evaluation methods either require expensive adversarial attacks or report only a single aggregate score that obscures how robustness is distributed across classes. We introduce the GF-Score (GREAT-Fairness Score), a framework that decomposes the certified GREAT Score into per-class robustness profiles and quantifies their disparity through four metrics grounded in welfare economics: the Robustness Disparity Index (RDI), the Normalized Robustness Gini Coefficient (NRGC), Worst-Case Class Robustness (WCR), and a Fairness-Penalized GREAT Score (FP-GREAT). The framework further eliminates the original method’s dependence on adversarial attacks through a self-calibration procedure that tunes the temperature parameter using only clean accuracy correlations. Evaluating 22 models from RobustBench across CIFAR-10 and ImageNet, we find that the decomposition is exact, that per-class scores reveal consistent vulnerability patterns (e.g., “cat” is the weakest class in 76% of CIFAR-10 models), and that more robust models tend to exhibit greater class-level disparity. These results establish a practical, attack-free auditing pipeline for diagnosing where certified robustness guarantees fail to protect all classes equally. We release our code on GitHub . 1 Introduction Deep neural networks are vulnerable to adversarial examples: imperceptible perturbations that cause confident misclassifications ( 29 ; 10 ) . This vulnerability poses serious risks in safety-critical settings such as autonomous driving and medical diagnosis, where a single misclassification can have catastrophic consequences. Adversarial training ( 19 ) remains the dominant defense, and substantial progress has been tracked through standardized benchmarks like RobustBench ( 7 ) , which ranks models by their accuracy under the AutoAttack ensemble ( 8 ) . Yet a fundamental tension exists between robustness and standard accuracy ( 31 ) , and recent work on certified defenses ( 5 ) has shifted attention toward provable guarantees rather than empirical attack evaluations alone. A critical limitation of current evaluation practice is that robustness is almost always reported as a single aggregate number. Whether the metric is empirical robust accuracy or a certified lower bound, it averages over the entire test distribution and thereby conceals how robustness is distributed across classes. Several studies have shown that adversarial training induces pronounced class-wise performance gaps: certain classes become far more vulnerable than others under the same model ( 3 ; 36 ; 30 ) . For instance, an autonomous perception system may appear globally robust while being nearly defenseless on pedestrian classes. Despite this, no existing framework provides certified, attack-free, per-class robustness evaluation. The GREAT Score ( 17 ) , introduced at NeurIPS 2024, offers a certified global robustness bound using only generative model samples and forward passes, achieving roughly 2,000 \times speedup over attack-based methods. However, it reports only a single scalar, inheriting the same class-blindness problem. Concurrently, training-time fairness interventions ( 32 ; 28 ; 15 ; 38 ) address the disparity during model optimization but offer no tools for post-hoc auditing of already-deployed models. In this paper, we introduce the GF-Score (GREAT-Fairness Score), a framework that bridges this gap through three components. First, we decompose the GREAT Score into per-class certified robustness profiles by partitioning samples according to their ground-truth labels and computing class-conditional confidence margins. This decomposition is exact: the weighted sum of per-class scores recovers the aggregate score with zero numerical error. Second, we quantify the disparity of these per-class profiles through four metrics grounded in welfare economics and fairness theory: the Robustness Disparity Index (RDI), the Normalized Robustness Gini Coefficient (NRGC), Worst-Case Class Robustness (WCR), and a Fairness-Penalized GREAT Score (FP-GREAT). Third, we eliminate the original method’s dependence on adversarial attacks for temperature calibration by introducing a self-calibration procedure that maximizes rank correlation with publicly available clean accuracies. We evaluate the GF-Score on 22 robust models from RobustBench spanning CIFAR-10 (17 \ell_{2} models) and ImageNet (5 \ell_{\infty} models). Our experiments yield several notable findings. The class-conditional decomposition is exactly consistent across all 22 models, confirming the mathematical validity of the approach. Per-class analysis reveals that the class “cat” is the most vulnerable in 76% of CIFAR-10 models, while “automobile” is consistently the most robust, suggesting that class vulnerability is an intrinsic data property rather than a training artifact. We observe a positive correlation between aggregate robustness and the Robustness Disparity Index, providing new quantitative evidence for the tension between robustness and fairness identified by prior work ( 36 ; 3 ) . Our attack-free self-calibration achieves a Spearman rank correlation of \rho=0.871 on CIFAR-10 and \rho=1.000 on ImageNet with RobustBench rankings, making the entire evaluation pipeline truly attack-free. Figure 1: Per-class GREAT Scores for 17 CIFAR-10 models reveal substantial class-level robustness disparity hidden by aggregate scores. Each row is a model (sorted by aggregate GREAT Score); each column is a class. The class “cat” is consistently the most vulnerable (darkest column), while “automobile” is most often the most robust (10 of 17 models). Models with higher aggregate robustness (top rows) tend to exhibit greater disparity between their strongest and weakest classes. In summary, we make the following contributions: 1. We propose a class-conditional decomposition of the GREAT Score that preserves the certified lower-bound guarantee at per-class granularity, with formal concentration bounds (Propositions 1 and 2 ). 2. We introduce four fairness-aware disparity metrics (RDI, NRGC, WCR, FP-GREAT) grounded in welfare economics that quantify how robustness is distributed across classes. 3. We propose an attack-free self-calibration procedure that replaces adversarial-attack-based temperature tuning with clean accuracy correlation, enabling fully attack-free evaluation. 4. We conduct extensive experiments on 22 models across two benchmarks , revealing consistent class vulnerability patterns and a quantifiable robustness-fairness tension that aggregate metrics conceal. 2 Related Work Our work lies at the intersection of three active research areas: adversarial robustness evaluation, fairness in adversarial training, and inequality measurement in machine learning. We synthesize each area below and position our contribution relative to existing methods in Table 1 . 2.1 Adversarial Robustness Evaluation Since the discovery that neural networks are vulnerable to imperceptible perturbations ( 29 ; 10 ) , a rich line of work has developed increasingly powerful attacks to evaluate robustness. Gradient-based methods such as PGD ( 19 ) and the C&W attack ( 4 ) became standard tools, though 2 showed that many defenses merely obfuscated gradients rather than achieving true robustness. The AutoAttack ensemble ( 8 ) addressed this by combining complementary attack strategies into a reliable evaluation protocol, and RobustBench ( 7 ) standardized model comparison through a public leaderboard. In parallel, certified approaches emerged to provide provable robustness guarantees. Convex relaxation methods ( 33 ) and randomized smoothing ( 5 ; 14 ; 25 ) offer formal certificates that no perturbation within a given norm ball can change the prediction. 16 provide a comprehensive systematization of these approaches. However, all certified methods operate at the per-sample level, producing local guarantees that must be aggregated to characterize a model’s overall robustness. The GREAT Score ( 17 ) took a fundamentally different approach by defining a global certified robustness metric over the data distribution using generative models, achieving strong rank correlation with RobustBench while requiring only forward passes. Our work extends the GREAT Score by decomposing this global metric into per-class

Read the complete text