BackFed: An Efficient & Standardized Benchmark Suite for Backdoor Attacks in Federated Learning

Thinh Dao, Dung Thuy Nguyen, Khoa D Doan, Kok-Seng Wong · 2025

Read BackFed: An Efficient & Standardized Benchmark Suite for Backdoor Attacks in Federated Learning by Thinh Dao, Dung Thuy Nguyen, Khoa D Doan, Kok-Seng Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04903.txt.

Introduction

Thinh Dao, Dung Thuy Nguyen, Khoa Doan, and Kok-Seng Wong introduce BackFed in this 2025 paper, a benchmark for backdoor attacks in federated learning. In federated learning, many devices train a shared model without sharing raw data. A backdoor attack poisons that training so the model fails on cue.

Research on such attacks and defenses has grown fast, but the testing is a mess. Studies use mismatched settings and threat models that do not match real deployments. Worse, the authors' code review found bugs in the official code of several attacks that inflated their reported success. BackFed answers by placing attacks and defenses under one shared frame that mirrors real use. The benchmark stress-tests each method on three representative setups. Its aim is to show which attacks truly work, and which only looked strong on a narrow stage.

Catalog record

CREATORThinh Dao, Dung Thuy Nguyen, Khoa D Doan, Kok-Seng Wong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04903

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BackFed: An Efficient & Standardized Benchmark Suite for Backdoor Attacks in Federated Learning Thinh Dao, Dung Thuy Nguyen, Khoa D Doan, Kok-Seng Wong Abstract Federated Learning (FL) systems are vulnerable to backdoor attacks, where adversaries train their local models on poisoned data and submit poisoned model updates to compromise the global model. Despite numerous proposed attacks and defenses, divergent experimental settings, implementation errors, and unrealistic assumptions hinder fair comparisons and valid conclusions about their effectiveness in real-world scenarios. To address this, we introduce BackFed – a comprehensive benchmark suite designed to standardize, streamline, and reliably evaluate backdoor attacks and defenses in FL, with a focus on practical constraints. Our benchmark offers key advantages through its multi-processing implementation that significantly accelerates experimentation and the modular design that enables seamless integration of new methods via well-defined APIs. With a standardized evaluation pipeline, we envision BackFed as a plug-and-play environment for researchers to comprehensively and reliably evaluate new attacks and defenses. Using BackFed, we conduct large-scale studies of representative backdoor attacks and defenses across both Computer Vision and Natural Language Processing tasks with diverse model architectures and experimental settings. Our experiments critically assess the performance of proposed attacks and defenses, revealing unknown limitations and modes of failures under practical conditions. These empirical insights provide valuable guidance for the development of new methods and for enhancing the security of FL systems. Our framework is openly available at https://github.com/thinh-dao/BackFed . 1 Introduction Federated Learning (FL) has emerged as a promising framework for training machine learning models on decentralized datasets across multiple devices [ 13 , 21 ] . This paradigm addresses increasing privacy concerns by enabling collaborative model training without sharing sensitive data, making it ideal for applications where data cannot be centralized due to privacy regulations or practical limitations. However, the distributed nature of FL introduces the threats of backdoor attacks [ 2 ] , which occur when malicious participants poison training data with specific triggers that cause the model to misclassify particular inputs during inference. As FL gains traction in critical domains like healthcare, finance, and edge computing [ 17 , 48 ] , the study of backdoor attacks has attracted increasing attention, with numerous attack and defense methods emerging in an ongoing arms race. However, the rapid development of new techniques in federated learning (FL) has introduced significant inconsistencies in how concurrent works are evaluated. Many proposed methods are evaluated incompletely, as attacks are only tested against a limited set of defenses, and vice versa. The experimental settings of different methods lack uniformity in key hyperparameters such as the number of clients, the number of attackers, attack frequency, and the degree of data heterogeneity across clients. These discrepancies undermine the validity of research findings, hinder fair comparisons between approaches, and obstruct a unified understanding of progress in this area. Moreover, many studies neglect practical deployment constraints, such as communication overhead and computational cost, limiting the real-world applicability of their solutions. To address these issues, we introduce BackFed, a comprehensive benchmarking suite designed to standardize, streamline, and reliably evaluate backdoor attacks and defenses in federated learning (FL). Our framework makes the following key contributions: • Parallelized Training: BackFed enables multiprocessing across simulated clients with Ray package, significantly accelerating experimentation compared to existing frameworks. • Modular codebase design: BackFed is organized into clearly separated modules with well - defined APIs. This structure makes it straightforward to plug in new algorithms. • Standardized Evaluation Pipeline: With Hydra package, BackFed delivers a plug - and - play evaluation environment that scales effortlessly to large - scale studies. New attacks and defenses can be added with just a few lines, then comprehensively and reliably assessed under realistic and diverse FL conditions, revealing their true strengths and vulnerabilities. Leveraging BackFed, our systematic investigation uncovers critical factors influencing attack performance and exposes weaknesses in prevalent defense strategies. These insights facilitate the development of more robust and secure FL systems resistant to backdoor threats. 2 Background and Related Work Federated Learning . In an FL system, each client i holds a local dataset D_{i} , and collaboratively trains a shared global model without exposing their data. The training objective is to minimize the sum of the local empirical losses \sum_{i=1}^{N}\mathcal{L}_{i}(D_{i};\theta) , where \mathcal{L}_{i} denotes the training objective of client i over D_{i} , and \theta is the global model. In each global round t , the central server broadcasts the current global model \theta^{t} to K randomly selected clients. Each selected client i locally trains its model parameters \theta^{t}_{i} by minimizing the local loss L_{i} on its private dataset D_{i} and then returns the updated parameters back to the server. Upon receiving updates from all participating clients, the server aggregates these local updates to produce the updated global model according to: \theta^{t+1}=\theta^{t}-\eta\cdot\text{AGR}\left(\theta^{t}_{1},\theta^{t}_{2},\dots,\theta^{t}_{K}\right), (1) where \eta is the server’s learning rate, and \text{AGR}(\cdot) represents the aggregation function to get the global model for the next round. In the baseline algorithm FedAvg, the aggregation function is a simple average: \text{AGR}(\cdot)=\frac{1}{K}\sum_{i}^{K}\Delta_{i}^{t}\text{ where }\Delta_{i}^{t}=\theta_{i}^{t}-\theta^{t} . We note that this function is a version of FedAvg introduced by [ 2 ] that assigns an equal weight \frac{1}{K} to all the local models instead of weighting them based on their dataset sizes [ 21 ] . This prevents malicious clients from exaggerating their dataset sizes to increase their influence on the global model. 2.1 Taxonomy of Backdoor Attacks in FL Taxonomy of Backdoor Attacks in FL. Backdoor attacks in FL can be broadly categorized into Data-Poisoning and Model-Poisoning attacks. Data-poisoning attacks inject malicious samples with specific triggers into the local dataset to implant backdoors in the global model. Various strategies exist depending on the design and distribution of the triggers, including Fixed-Pattern, Dynamic-Pattern, Distributed-Pattern, and Edge-Case attacks [ 2 , 8 , 41 , 38 , 33 ] . In contrast, model-poisoning attacks directly manipulate the training process or model updates to increase the effectiveness of the backdoor or to evade defenses. Representative examples include Model-Replacement attacks that scale malicious updates, durability-enhanced techniques such as Neurotoxin and Chameleon [ 45 , 9 ] , defense-evasion approaches like Constrain-and-Scale, 3DFed, and Backdoor-Critical Layers [ 14 , 47 ] . Taxonomy of Backdoor Defenses in FL. Backdoor defenses in FL can be categorized based on the stage of deployment. Client-side defenses , such as FL-WBC, FLIP, and LocalDP [ 30 , 43 , 22 ] , aim to improve the robustness of local training or introduce noise to client updates. Pre-aggregation defenses focus on detecting and filtering out suspicious updates before aggregation using anomaly detection techniques, as in DeepSight, FLAME, BackdoorIndicator, and FLDetector [ 32 , 27 , 23 , 15 , 44 ] . In-aggregation defenses modify the server’s aggregation rule to reduce the influence of malicious or outlier updates, with approaches such as Median, RFA, Trimmed Mean, Krum, FoolsGold, FLTrust, and Robust Learning Rate (RLR) [ 40 , 4 , 10 , 25 , 7 ] . Post-aggregation defenses intervene after the global model has been aggregated, aiming to suppress or eliminate backdoors that may persist. Notable examples include WeakDP [ 34 ] and CRFL [ 39 ] , which clip the model’s weight norm and inject noise as a form of regularization that improves model

Read the complete text