Generalization Certificates for Adversarially Robust Bayesian Linear Regression

Mahalakshmi Sabanayagam, Russell Tsuchida, Cheng Soon Ong, Debarghya Ghoshdastidar · 2025

Read Generalization Certificates for Adversarially Robust Bayesian Linear Regression by Mahalakshmi Sabanayagam, Russell Tsuchida, Cheng Soon Ong, Debarghya Ghoshdastidar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14298.txt.

Introduction

Mahalakshmi Sabanayagam and three co-authors derived in 2025 the first guarantees that a Bayesian regression model will stay reliable under attack. Attackers nudge inputs to flip predictions, and most defenses cover only single-point answers, not models that give probability ranges. The team recast the attack as a cost function with a tidy geometry.

That geometry yields the worst nudge in closed form and a new kind of hardened posterior belief. A general theory then certifies how far the model's accuracy can fall. Experiments on real and synthetic data back the math.

Catalog record

CREATORMahalakshmi Sabanayagam, Russell Tsuchida, Cheng Soon Ong, Debarghya Ghoshdastidar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14298

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generalization Certificates for Adversarially Robust Bayesian Linear Regression Mahalakshmi Sabanayagam, Russell Tsuchida, Cheng Soon Ong, Debarghya Ghoshdastidar Abstract Adversarial robustness of machine learning models is critical to ensuring reliable performance under data perturbations. Recent progress has been on point estimators, and this paper considers distributional predictors. First, using the link between exponential families and Bregman divergences, we formulate an adversarial Bregman divergence loss as an adversarial negative log-likelihood. Using the geometric properties of Bregman divergences, we compute the adversarial perturbation for such models in closed-form. Second, under such losses, we introduce adversarially robust posteriors , by exploiting the optimization-centric view of generalized Bayesian inference. Third, we derive the first rigorous generalization certificates in the context of an adversarial extension of Bayesian linear regression by leveraging the PAC-Bayesian framework. Finally, experiments on real and synthetic datasets demonstrate the superior robustness of the derived adversarially robust posterior over Bayes posterior, and also validate our theoretical guarantees. 1]School of Computation, Information and Technology Technical University of Munich Germany 2]Munich Data Science Institute Technical University of Munich Germany 3]Monash University 4 Data61, CSIRO 5 College of Systems and Society Australian National University Australia ] [email protected], [email protected], [email protected], [email protected] 1 Introduction Machine learning models are vulnerable to adversarial inputs, where small, carefully crafted perturbations to the input data can significantly degrade model performance. These perturbations, though imperceptible to humans (e.g. in computer vision contexts), can cause models to make incorrect predictions with high confidence. Significant progress has been made in understanding and improving adversarial robustness of point predictors, with efforts in defense mechanisms, attack strategies, and the trade-offs between robustness and generalization [ Szegedy et al. 2014 , Shafahi et al. 2019 , Li et al. 2023 ] . A key insight from this body of research is that models susceptible to adversarial attacks often exhibit near-perfect empirical generalization –— achieving similar performance on training and test data [ Goodfellow et al. 2015 ] .This observation suggests that deriving formal guarantees is critical to understanding the interplay between generalization and adversarial robustness. Probabilistic models offer an alternative paradigm that quantifies uncertainty, a property that can detect adversarial inputs and reject uncertain predictions. However, despite these advantages, probabilistic models have received significantly less attention than their non-probabilistic counterparts in adversarial settings [ Bradshaw et al. 2017 , Grosse et al. 2018 ] . While they have been studied for robustness against outliers [ Kim and Ghahramani 2008 ] , label noise [ Hernández-Lobato et al. 2011 ] , and domain shifts [ Ovadia et al. 2019 ] , their susceptibility to adversarial attacks remains largely unexplored. There exists no notion of adversarially robust probabilistic inference, and hence no formal generalization guarantees, thus raising a fundamental question: How can we define and develop adversarially robust probabilistic inference and derive generalization certificates (formal guarantees) for such models? In this work, we address this question by first introducing the notion of adversarially robust posteriors . We achieve this by formulating an adversarial variant of the negative log-likelihood (NLL) loss — drawing inspiration from adversarial training, one of the most effective defense strategies against adversarial attacks in standard machine learning [ Madry et al. 2018 ] — and taking an optimization-centric perspective of (generalized) Bayesian inference [ Alquier et al. 2016 ] . In doing so, we obtain a posterior that is robust to adversarial perturbations. We leverage the PAC-Bayesian framework, a powerful tool for deriving data-dependent generalization bounds for Bayesian predictors [ McAllester 1998 , Catoni and Picard 2004 ] , and derive the first rigorous certificates for the robust posterior on linear regression. Furthermore, we also derive PAC-Bayesian based generalization certificates for Bayes posterior obtained using the standard negative log-likelihood loss. In summary, our main contributions are as follows. Standard generalization Adversarial generalization NLL \ell Adversarial NLL \ell_{\hat{\delta}} R(\theta)=\underset{\mathcal{D}\sim\mathcal{P}}{\mathbb{E}}\left[\ell(\theta,\mathcal{D})\right] R_{\hat{\delta}}(\theta)=\underset{\mathcal{D}\sim\mathcal{P}}{\mathbb{E}}\left[\ell_{\hat{\delta}}(\theta,\mathcal{D})\right] Bayes posterior q \underset{\theta\sim q}{\mathbb{E}}\left[R(\theta)\right] \underset{\theta\sim q}{\mathbb{E}}\left[R_{\hat{\delta}}(\theta))\right] Theorem 4.3 Theorem 4.4 Robust posterior q_{\delta} \underset{\theta\sim q_{\delta}}{\mathbb{E}}\left[R(\theta)\right] \underset{\theta\sim q_{\delta}}{\mathbb{E}}\left[R_{\hat{\delta}}(\theta)\right] Theorem 4.5 Theorems 4.6 and 4.7 Table 1: Overview of our derived generalization certificates. The guarantees are derived for the standard Bayes posterior q and the novel robust posterior q_{\delta} , where \delta denotes the training adversarial allowance. We consider generalization to standard NLL loss ( standard generalization ) as well as adversarial NLL ( adversarial generalization ), with a potentially different adversarial allowance \hat{\delta} . [( i ), topsep=0pt,itemsep=-1ex,partopsep=1ex,parsep=1ex] 1. In exponential families, we review a one-to-one correspondence between the adversarial negative log-likelihood loss and the class of Bregman divergences. Based on this correspondence, we introduce a novel adversarial negative log-likelihood loss in Section 3 . This probabilistic-geometric connection allows us to solve the adversarial perturbation problem in closed-form, for all exponential families, allowing for an adversarially robust formulation of generalized linear models. 2. We define the adversarially robust posterior as minimizing a variational objective with this loss, thereby extending Bayesian inference to adversarially robust generalized linear models settings, in Section 3 . 3. In Section 4 , focusing on the case of a Gaussian family, we derive the PAC-Bayesian generalization certificates for the Bayes posterior and the robust posterior under two settings: a) standard generalization : guarantees for standard negative log-likelihood loss ( Theorems 4.3 and 4.4 ), and b) adversarial generalization : guarantees for adversarial negative log-likelihood loss ( Theorems 4.5 , 4.6 and 4.7 ). Table 1 gives an overview of our bounds. We experimentally validate the derived certificates in Section 5 showing non-trivial guarantees. We discuss the practical significance of the bounds, several technicalities, and touch on related works in Section 6 ; and conclude in Section 7 . 2 Preliminaries Our work combines elements from adversarial robustness and probabilistic inference. We briefly outline these topics here, and establish some preliminaries. Notation We represent the entry-wise absolute value of matrix M as |M| , and vector Euclidean norm as \|\cdot\| . We use I_{n} for identity matrix of size n\times n and 1_{n} for a vector of size n with all ones. We denote by \ell\big(\theta,(x,y)\big) a loss evaluated on parameter \theta and single (x,y) data pair, and use \mathcal{L}(\theta,\mathcal{D})=\sum_{i=1}^{n}\ell\big(\theta,(x_{i},y_{i})\big) for the sum of the losses over a dataset. The expected and empirical average errors are R(\theta)=\underset{(x,y)\sim\mathcal{P}}{\mathbb{E}}\left[\ell\big(\theta,(x,y)\big)\right] and r(\theta)=\frac{1}{n}\mathcal{L}(\theta,\mathcal{D}) . 2.1 Adversarial robustness We are given n labeled data samples drawn i.i.d. from an unknown probability measure \mathcal{P} , denoted by \mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{n} with x_{i}\in\mathbb{X}\subseteq\mathbb{R}^{d} representing the feature vector with the corresponding label y_{i}\in\mathbb{R} . We later use X\in\mathbb{R}^{n\times d} to refer to the matrix of all feature vectors, and Y\in\mathbb{R}^{n} for the vector of all n labels. We follow the standard setting of supervised learning [ Bishop 2007 , Deisenroth et al. 2020 ] , where we minimize the empirical loss \ell on the training set \mathcal{D}=(X,Y)=\{x_{i},y_{i}\}_{i=1}^{n} with respect to the parameters \theta of our model, \displaystyle\theta^{\ast}=\argmin_{\theta}\sum_{i=1}^{n}\ell\big(\theta,(x_{i},y_{i})\big)=\argmin_{\theta}\mathcal{L}(\theta,\mathcal{D}). For the adversarially robust setting, we consider perturbed data \widetilde{x}_{i} but not perturbed labels. Following typical adversarial constructs [ Szegedy et al. 2014 , for example] we consider perturbations whose \ell_{2} distance is bounded by a user-defined constant \delta . \displaystyle\theta^{\ast}=\argmin_{\theta}\sum_{i=1}^{n}\max\limits_{\|\widetilde{x}_{i}-x_{i}\|\leq\delta}\ell\big(\theta,(\widetilde{x}_{i},y_{i})\big). Here we focus on the parametric supervised setting, where a parameter \theta is mapped to a prediction f_{\theta}(x_{i}) on a feature x_{i} with some parameterized function f_{\theta}:\mathbb{X}\to\mathbb{F} . 2.2 Probabilistic inference Bayesian inference Of central interest in

Read the complete text