Identifying Adversary Characteristics from an Observed Attack

Soyon Choi, Scott Alfeld, Meiyi Ma · 2026

Read Identifying Adversary Characteristics from an Observed Attack by Soyon Choi, Scott Alfeld, Meiyi Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05625.txt.

Introduction

Choi and team look at guarding against data-tamper attacks by studying the foe, not the attack itself. They present a framework for naming foe traits from an observed attack. Lacking extra knowledge, the foe cannot be named, since several foes could produce the same observed attack. A field-free framework then names the most likely foe. Foe knowledge aids defense twice: it enables fixes outside the learning code or limits foe power. It also improves defense methods that hit learning head-on, such as hostile rule-setting.

Catalog record

CREATORSoyon Choi, Scott Alfeld, Meiyi Ma
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72186
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05625

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Identifying Adversary Characteristics from an Observed Attack Soyon Choi, Scott Alfeld, Meiyi Ma Abstract When used in automated decision-making systems, machine learning (ML) models are vulnerable to data-manipulation attacks. Some defense mechanisms (e.g., adversarial regularization) directly affect the ML models while others (e.g., anomaly detection) act within the broader system. In this paper we consider a different task for defending the adversary, focusing on the attack er , rather than the attack. We present and demonstrate a framework for identifying characteristics about the attacker from an observed attack. We prove that, without additional knowledge, the attacker is non-identifiable (multiple potential attackers would perform the same observed attack). To address this challenge, we propose a domain-agnostic framework to identify the most probable attacker. This framework aids the defender in two ways. First, knowledge about the attacker can be leveraged for exogenous mitigation (i.e., addressing the vulnerability by altering the decision-making system outside the learning algorithm and/or limiting the attacker’s capability). Second, when implementing defense methods that directly affect the learning process (e.g., adversarial regularization), knowledge of the specific attacker improves performance. We present the details of our framework and illustrate its applicability through specific instantiations on a variety of learners. 1 Introduction Despite their capabilities, machine learning (ML) systems are vulnerable to adversarial attacks — imperceptible perturbations to input data that cause the model to predict incorrectly 18 ; 15 ; 9 . To counter known attack strategies, many methods of defense have been proposed 4 ; 8 . However, these defenses assume a specific attacker, and are often easily circumvented by more sophisticated attackers, leading to a constant arms race between new attack strategies and new defense strategies 20 . The foundational weakness of many existing defenses is that they typically operate under fixed threat models that assume the attacker’s parameters, including their level of knowledge, capabilities/limitations, and objectives. While this is analytically convenient, such assumptions rarely reflect the behavior of real adversaries, whose parameters are non-stationary and/or unknown. This motivates two distinct but complementary goals. First, we want to better understand the attacker itself. In real-life ML applications, learning about the knowledge, objectives, and capabilities of the attacker provides insight into the nature of the overall threat. The importance of “knowing one’s enemy” has long been recognized in adversarial settings 21 , and in the ML context, such knowledge can inform risk assessment, attack detection, and robust design of future systems. Hence, this is valuable even independent of any particular defense mechanism. Second, the knowledge of the attacker can be used toward a tailored defense. In practice, the attacker’s beliefs, objective, and capabilities are almost never known by the defender; the only information that is reliably known by the defender is the actual input to the defender’s model. Hence, rather than assuming fixed attacker parameters, the defender should infer them from the attacks they observe and adapt their defensive strategy accordingly. For example, the defender can apply adversarial regularization under the attack parameters inferred from the observed attacks. In this paper, we propose a novel, domain-agnostic framework for reverse engineering the goals and parameters of the attacker. Within our novel framework, we formulate the defender’s task as a reverse optimization problem that outputs the most likely attacker parameters given an observed attack and a prior belief distribution. We then provide a simple proof-of-concept of the feasibility of our framework. Concretely, we make the following contributions: 1. We introduce a general framework and method for reverse-engineering aspects of an attacker from their attack. 2. We mathematically illustrate the limitations of the defender by showing that, in general, the aspects of an attacker are non-identifiable. 1.1 Mathematical Notation Throughout this paper, we use bold uppercase to denote matrices, bold lowercase to denote vectors, and plain lowercase to denote scalars. Special characters on recurring variables are used to differentiate between steps of the pipeline. We decorate with hat ( \hat{\cdot} ) to denote a learned variable and star ( \cdot^{*} ) to denote its ground-truth value. For instance, consider a variable \boldsymbol{A} . \hat{\boldsymbol{A}} is the learned \boldsymbol{A} , i.e., the result of an optimization problem over \boldsymbol{A} . \boldsymbol{A}^{*} is the true value of \boldsymbol{A} . All variables used are defined within the paper, and a full notation table is provided for reference in Appendix A. 2 Threat Model We consider two players, the defender/learner DFDR and the attacker/adversary ATKR . ATKR performs a deployment-time data-manipulation attack against DFDR . DFDR ’s goal is to learn about ATKR from the observed attack. Based on the classical taxonomy widely used in adversarial machine learning 6 ; 5 ; 13 , we model ATKR as having three components: 1. \mathcal{K} : Their knowledge about DFDR . What does ATKR think DFDR ’s prediction function is? 2. \mathcal{C} : Their capability. What perturbations to the data can ATKR make? 3. \mathcal{O} : Their objective. What function does ATKR try to optimize? For ease of notation, we assume the knowledge, capability, and objective of ATKR are parameterized, and \mathcal{K},\mathcal{C},\mathcal{O} denote the sets of parameters that pertain to ATKR ’s knowledge, capability, and objective, respectively. ATKR observes the original input \boldsymbol{x} and aims to perform the optimal attack \boldsymbol{\alpha}_{opt} given their values of (\mathcal{K},\mathcal{C},\mathcal{O}) . We denote the optimal attack as a function of ATKR ’s parameters ( \boldsymbol{\alpha}_{opt}(\mathcal{K},\mathcal{C},\mathcal{O}) ). We assume that DFDR knows the parameterized form of ATKR . The task of DFDR is to, after observing \boldsymbol{\alpha}_{obs} , reverse engineer \mathcal{K},\mathcal{C},\mathcal{O} . This is achieved through our framework, which is introduced in Section 3. Figure 1 shows the overview of the ATKR - DFDR system, including the role of our framework within it. We focus on deployment-time (as opposed to training-time) attackers of two forms–those that perform “attractive” attacks (aiming to pull the prediction toward their particular goal) and “repulsive” attacks (aiming to push the prediction as far as possible from the original prediction). In the following subsections, we explore three different possible ATKR - DFDR configurations and their respective ATKR parameterizations: 1. Linear Regressor DFDR vs. Repulsive ATKR 2. Logistic Regressor DFDR vs. Attractive ATKR 3. MLP DFDR vs. Attractive ATKR These three examples result in qualitatively different mathematical forms of increasing difficulty, which we discuss in more detail in the next section. We note that the selected models as well as ATKR / DFDR parameterizations presented in this section are meant as illustrative examples. Our framework is domain-agnostic and freely admits other attackers, defenders, and parameterizations thereof. 2.1 Linear Regression Suppose DFDR is a generalized multi-target linear regressor with the prediction function f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{q} : \displaystyle f(\boldsymbol{x}) \displaystyle=\boldsymbol{M}^{*}\boldsymbol{x} (1) where \boldsymbol{M}^{*}\in\mathbb{R}^{q\times d} is the weight matrix of the model. Suppose ATKR performs a repulsive attack under Mahalanobis constraints . That is, ATKR aims to perturb the input within its capabilities such that DFDR ’s prediction is as wrong as possible. ATKR perturbs a test point \boldsymbol{x} by adding a perturbation vector \boldsymbol{\alpha} . ATKR aims to solve the following optimization problem: \displaystyle\boldsymbol{\alpha}_{opt}(\mathcal{K},\mathcal{C},\mathcal{O}) \displaystyle=\arg\max_{\boldsymbol{\alpha}}\|\boldsymbol{M}(\boldsymbol{x}+\boldsymbol{\alpha})-\boldsymbol{M}\boldsymbol{x}\|_{\boldsymbol{W}}^{2} (2) \displaystyle=\arg\max_{\boldsymbol{\alpha}}\|\boldsymbol{M}\boldsymbol{\alpha}\|_{\boldsymbol{W}}^{2} (3) \displaystyle\text{s.t.~}||\boldsymbol{\alpha}||_{\boldsymbol{C}}\leq c (4) where • \boldsymbol{M}\in\mathbb{R}^{q\times d} is ATKR ’s estimate of DFDR ’s model \boldsymbol{M}^{*} . • \boldsymbol{W}\in\mathbb{R}^{q\times q} is a positive definite matrix defining the Mahalanobis loss of ATKR . • \boldsymbol{C}\in\mathbb{R}^{d\times d} is

Read the complete text