Byzantine Resilient Federated Multi-Task Representation Learning

Tuan Le, Shana Moothedath · 2025

Read Byzantine Resilient Federated Multi-Task Representation Learning by Tuan Le, Shana Moothedath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19209.txt.

Introduction

Tuan Le and Shana Moothedath harden federated learning against saboteurs in a 2025 paper. Many clients train one shared representation, each keeping a private final layer. That split captures common features while letting each client adapt. Some clients, though, may be faulty or malicious, sending poisoned updates to the server.

Their framework alternates local training with careful aggregation, using two robust defenses against Byzantine agents. Personal models then survive a minority of bad actors. Federated setups in hospitals and phones cannot vet every participant, so resilience by design is the only scalable answer. The paper supplies it for the multi task representation case.

Catalog record

CREATORTuan Le, Shana Moothedath
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-19209

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Byzantine Resilient Federated Multi-Task Representation Learning Tuan Le, Shana Moothedath Abstract In this paper, we propose BR-MTRL, a Byzantine-resilient multi-task representation learning framework that handles faulty or malicious agents. Our approach leverages representation learning through a shared neural network model, where all clients share fixed layers, except for a client-specific final layer. This structure captures shared features among clients while enabling individual adaptation, making it a promising approach for leveraging client data and computational power in heterogeneous federated settings to learn personalized models. To learn the model, we employ an alternating gradient descent strategy: each client optimizes its local model, updates its final layer, and sends estimates of the shared representation to a central server for aggregation. To defend against Byzantine agents, we employ two robust aggregation methods for client-server communication, Geometric Median and Krum. Our method enables personalized learning while maintaining resilience in distributed settings. We implemented the proposed algorithm in a federated testbed built using Amazon Web Services (AWS) platform and compared its performance with various benchmark algorithms and their variations. Through experiments using real-world datasets, including CIFAR-10 and FEMNIST, we demonstrated the effectiveness and robustness of our approach and its transferability to new unseen clients with limited data, even in the presence of Byzantine adversaries. I Introduction Machine learning has traditionally relied on centralized settings, where a single model is trained on large datasets stored in one central location. This approach requires gathering data from all clients into a central data center, which raises significant data privacy concerns. Federated Learning (FL) addresses these challenges by using a distributed framework that preserves data privacy while overcoming the problem of isolated data sources [ 1 ] . FL enables multiple clients, such as mobile devices or institutions, to collaboratively train a global model without sharing their raw data. A central server is responsible for coordinating the clients by aggregating the models received from each client and broadcasting the updated model back to them in each round of communication. McMahan et al. introduced the Federated Averaging (FedAvg) algorithm, which facilitates the training of a single shared global model designed to achieve optimal performance on average across all participating clients [ 1 ] . The approach offers several advantages, such as simplicity in implementation and a streamlined aggregation process, making it scalable for large client populations. Additionally, the global model promotes uniformity, ensuring consistent performance for clients with similar data distributions. However, FL approaches encounter three primary bottlenecks: (i) Data heterogeneity: In real-world scenarios, client tasks often exhibit significantly different underlying data distributions. In such environments, a global model optimized for average loss may underperform across diverse clients, as it fails to account for these variations in data [ 2 ] . (ii) Generalization and adaptability: Federated models may struggle to generalize to unseen clients that were not part of the training process, limiting their effectiveness in dynamic, real-world applications [ 3 ] . (iii) Byzantine attacks: FL is particularly vulnerable to Byzantine attacks, as it relies on distributed client updates; a faulty or malicious client can potentially corrupt the global model, compromising performance and reliability [ 4 ] . To solve data heterogeneity, many personalized FL algorithms have been developed. Yuyang Deng et al. proposed the Adaptive Personalized Federated Learning (APFL) algorithm to solve the problem of statistical heterogeneity in federated learning by applying a mix of local and global model optimization [ 5 ] . This method allows clients to train personalized models that better generalize on local distributions while still contributing to the learning of a robust global model. In addition, Alireza Fallah et al. proposed a method called Personalized Federated Learning via Model-Agnostic Meta-Learning (MAML) to solve the problem of personalized adaptation in federated learning by applying a meta-learning framework [ 6 ] . This method allows clients to quickly adapt a shared initial model to their local data distributions by performing a few steps of gradient descent, leading to more personalized models for each user. To achieve generalizability and adaptability, Multi-Task Representation Learning (MTRL) has emerged as a promising methodology in machine learning, aiming to address multiple related tasks/clients simultaneously by leveraging shared representations [ 7 ] . The core principle of MTRL is based on the idea that, even though data in federated settings is often non-i.i.d. (non-independent and identically distributed) across clients, there exists a shared learned representation across multiple clients that effectively captures the common structural elements inherent to each client. This shared foundation enables the use of simple linear predictors in the final model layer, which are then fine-tuned to address the unique variations and requirements of each client. This approach significantly enhances the efficiency of model training processes, as it allows for the transfer and utilization of learned features across different but related tasks . Several works have studied MTRL to address different aspects of learning and model performance. Yu Xie et al. solved the problem of simultaneously optimizing node classification and link prediction tasks by applying a shared feature extraction module followed by client-specific modules [ 8 ] . Yang et al. proposed a deep MTRL framework using tensor factorization, which enables automatic knowledge sharing across all layers of a deep neural network. This method applies to both homogeneous and heterogeneous settings and eliminates the need for manual specification of multi-task sharing strategies [ 9 ] . In addition, recent studies [ 3 , 10 , 11 , 12 ] have focused on the theoretical understanding of MTRL, particularly when the shared model exhibits specific structures, such as a low-rank representation. To address the challenge of Byzantine attacks in federated learning, Gouissem et al. developed a collaborative cross-check mechanism that enhances the system’s resilience by allowing clients to collaboratively identify malicious nodes based on each other’s contributions to model updates [ 13 ] . While effective, this approach introduces significant computational overhead due to the extensive cross-validation among clients, which can lead to inefficiencies. In contrast, Feng Lin et al. tackled Byzantine resilience using robust aggregation techniques, such as geometric median, Krum, and h-Krum, to address the matrix completion problem while ensuring resilience against Byzantine failures [ 14 ] . Recently, Li et al. studied Byzantine resilient FL for non-iid data by obtaining an inversed artificial gradient [ 4 ] . Zhang et al. investigated communication-efficient approaches for Byzantine-resilient FL and proposed a projected stochastic gradient descent (SGD) algorithm to enhance system robustness [ 15 ] . Their approach employed a Huber function-based robust aggregation with an adaptive threshold selecting strategy at the server to reduce the effects of Byzantine attacks. Despite these advancements, a unified approach for Byzantine-resilient personalized FL with strong generalization remains unavailable. In this paper, we introduce the Byzantine-resilient Multi-Task Representation Learning (BR-MTRL) framework to jointly address three critical challenges in FL: data heterogeneity, generalization, and Byzantine resilience. By incorporating geometric median and Krum aggregation methods into personalized FL, we enhance the global model’s resilience against Byzantine attacks. Our framework ensures that the models learned across distributed agents remain robust, reliable, and secure, even in the presence of adversarial influences, improving both performance and safety. I-A Contributions The contributions of this paper are threefold. • We propose a Byzantine-resilient MTRL algorithm based on personalized federated learning.

Read the complete text