On the Escaping Efficiency of Distributed Adversarial Training Algorithms
Ying Cao*, Kun Yuan, Ali H. Sayed · 2025
Read On the Escaping Efficiency of Distributed Adversarial Training Algorithms by Ying Cao*, Kun Yuan, Ali H. Sayed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-11337.txt.
Introduction
Ying Cao, Kun Yuan, and Ali H. Sayed compare ways of training models to resist adversarial attacks across a network of agents in this 2025 machine learning paper. Robustness is tied to how flat the found minimum is, so the authors study how quickly different training schemes escape sharp local minima.
Their theoretical framework covers centralized training and two decentralized styles, consensus and diffusion. When attacks are mild and batches are large, the decentralized schemes provably escape local minima faster, drifting toward flatter, more robust solutions. As the allowed attack strength grows, that advantage can disappear. Simulations back the theory and systematically compare the robustness of models from each scheme, making a case for decentralized training in distributed deployments.
Catalog record
| CREATOR | Ying Cao*, Kun Yuan, Ali H. Sayed |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 138 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-11337
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Escaping Efficiency of Distributed Adversarial Training Algorithms Ying Cao*, Kun Yuan, Ali H. Sayed Abstract Adversarial training has been widely studied in recent years due to its role in improving model robustness against adversarial attacks. This paper focuses on comparing different distributed adversarial training algorithms–including centralized and decentralized strategies–within multi-agent learning environments. Previous studies have highlighted the importance of model flatness in determining robustness. To this end, we develop a general theoretical framework to study the escaping efficiency of these algorithms from local minima, which is closely related to the flatness of the resulting models. We show that when the perturbation bound is sufficiently small (i.e., when the attack strength is relatively mild) and a large batch size is used, decentralized adversarial training algorithms–including consensus and diffusion–are guaranteed to escape faster from local minima than the centralized strategy, thereby favoring flatter minima. However, as the perturbation bound increases, this trend may no longer hold. In the simulation results, we illustrate our theoretical findings and systematically compare the performance of models obtained through decentralized and centralized adversarial training algorithms. The results highlight the potential of decentralized strategies to enhance the robustness of models in distributed settings. I Introduction Despite their success on a wide range of modern vision and language tasks, machine learning models have been shown to be surprisingly vulnerable to adversarial perturbations [ 1 , 2 ] . Basically, models that perform well under standard (clean) conditions can often be misled by imperceptible input modifications. One of the most widely used methodologies to mitigate this vulnerability is adversarial training [ 3 , 4 , 5 ] , where adversarially perturbed samples are included during training to improve the robustness of models. Earlier works usually studied adversarial robustness in the single-agent setting, where the entire system is composed of a single model. However, distributed learning systems, which involve multiple agents, are becoming increasingly relevant in highly connected world and for large-scale applications [ 6 , 7 ] . In the previous work [ 8 ] , we proposed a general distributed adversarial training framework to improve the robustness of networked models and analyzed the convergence behavior of the associated algorithms. In distributed learning, a group of agents collaboratively optimize a global objective. To achieve this, two main popular methodological paradigms are commonly employed in algorithm design [ 9 , 10 ] . The first is known as the centralized approach, in which all agents transmit their local observations to a central server that performs the global computation. This approach has theoretical properties that are essentially consistent with traditional single-agent learning, since the central server aggregates and processes all information. The second is the decentralized strategy, which differs fundamentally from the centralized setting: all agents are connected by a predefined graph topology, and each agent keeps its local observations private and interacts only with its neighboring agents to update its model. Furthermore, within the decentralized setting, diffusion and consensus are two widely adopted strategies [ 6 , 9 , 10 , 11 , 12 ] and are also the focus of this paper. The decentralized framework is primarily motivated by advantages in preserving data privacy, improving fault tolerance, and distributing the computational load across multiple agents [ 9 , 13 , 14 , 15 , 16 , 17 , 18 ] . Regarding the fundamental performance analysis in the standard (clean) environment, early studies generally established that decentralized methods can, at best, match the optimization performance of the centralized approach [ 9 , 19 , 20 , 21 , 22 , 23 ] . Some works such as [ 24 , 25 ] suggested that the decentralized operation may even degrade the generalization performance of models relative to the centralized baseline. However, some recent studies have observed that decentralized training can be beneficial to generalization [ 26 , 27 , 28 , 29 ] . Specifically, reference [ 26 ] empirically found that the consensus distance–i.e., the difference among local models during intermediate stages of training with decentralized strategies–could improve generalization over the centralized training. Similarly, [ 27 ] established that poorly-connected graphs can be beneficial to generalization. Similarly, reference [ 28 , 29 ] provably verified that decentralized training methods favor flatter local minima than the centralized approach in non-convex and large-batch settings, and such flatness has been linked to enhanced generalization ability [ 30 ] . In particular, [ 29 ] showed that decentralized strategies generally strike a more favorable balance between flatness and optimization compared to the centralized solution, thereby resulting in better classification accuracy. Motivated by these observations, this work aims to systematically compare decentralized and centralized adversarial training strategies where adversarial attacks are accounted for. In the adversarial learning community, several prior works have highlighted the importance of the flatness of models in determining robustness [ 31 , 32 , 33 , 34 ] . Specifically, they found that flattening the loss landscape with respect to the model parameters tends to improve the robustness of models. Inspired by these findings, we investigate the implicit bias of decentralized and centralized adversarial training methods, particularly regarding their preference for flatter minima and the impact on model robustness. In particular, we focus on addressing the following two challenges: • In the adversarial training context, can decentralized methods favor flatter models than the centralized strategy, and under what conditions? • Can decentralized adversarial training methods enhance robustness of models to adversarial attacks compared to their centralized counterpart? To answer these questions, we theoretically examine the escaping efficiency of the distributed adversarial training algorithms, which measures how effectively they escape local minima in non-convex risk landscapes. This efficiency is closely related to their preference for flatter solutions. The key contributions of this work are summarized as follows: (1) To analyze the escaping efficiency of distributed adversarial training algorithms, we extend the theoretical framework from the clean environment [ 29 ] to the adversarial learning context. Since the dependence of the Hessian matrix of the risk function on the iterates now renders the analysis rather intractable, we adopt a short-term model in which the original Hessian matrix is approximated by it value at a local minimum. To ensure that the short-term approximation reliably captures the dynamics of the true model in the vicinity of local minima, we rigorously verify that the approximation error between the true dynamics and the short-term model is negligible. It is important to note that this is not a straightforward extension, as the adversarial learning context requires appropriate modifications to the proof techniques due to changes in certain theoretical properties. For instance, the Lipshictz continuity of the risk function may no longer hold, and the presence of adversarial perturbations introduces additional complexities that must be carefully addressed. (2) Using the short-term model, we derive closed-form expressions for the escaping efficiency of distributed adversarial training algorithms, enabling a direct comparison of centralized, consensus, and diffusion strategies in the adversarial setting. Basically, our analysis confirms that the network heterogeneity and underlying graph structure contribute to faster escape from local minima for the family of decentralized adversarial training methods–namely, consensus and diffusion–compared to the centralized strategy. As a result, decentralized methods favor flatter minima than the centralized approach in