Understanding Communication Backends in Cross-Silo Federated Learning

Amir Ziashahabi † , Chaoyang He ‡ , Salman Avestimehr † · 2026

Read Understanding Communication Backends in Cross-Silo Federated Learning by Amir Ziashahabi † , Chaoyang He ‡ , Salman Avestimehr † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10859.txt.

Introduction

Ziashahabi, He, and Avestimehr test message-passing backends for cross-silo federated learning, where strong servers team up over varied links and big models strain the net. Their tests cover MPI, gRPC, and PyTorch RPC for single-link and end-to-end speed across model sizes under lifelike net conditions. They also add gRPC+S3, a hybrid backend fixing current limits for big-model transfer across far-flung sites, up to 3.8 times faster end to end than gRPC. The results guide backend choice and setup for given tasks and nets.

Catalog record

CREATORAmir Ziashahabi † , Chaoyang He ‡ , Salman Avestimehr †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23841
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10859

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Communication Backends in Cross-Silo Federated Learning Amir Ziashahabi † , Chaoyang He ‡ , Salman Avestimehr † Abstract Federated learning (FL) has emerged as a practical means for privacy-preserving distributed machine learning. FL’s versatile design makes it suitable for various training settings, from IoT edge devices in cross-device FL to powerful servers in cross-silo FL. A key consequence of this versatility is the high level of diversity found in the networking configuration of FL applications. Coupled with the rising demand for large-scale models such as large language models, well-informed selection and configuration of communication backends become crucial for ensuring optimal performance in FL systems. This work focuses on cross-silo federated learning, presenting in-depth benchmarks of various communication backends, including MPI, gRPC, and PyTorch RPC. In addition, we introduce gRPC+S3, a hybrid backend designed to overcome the limitations of existing approaches, particularly for transmitting large models across geo-distributed deployments, achieving up to 3.8 × 3.8\times end-to-end speedup over gRPC. Our benchmarks examine point-to-point and end-to-end performance for a broad range of model sizes running under realistic network conditions. Our findings provide practical insights for selecting and configuring suitable communication backends tailored to the specific federated learning tasks and network configurations. I Introduction Cross-silo federated learning (FL) is a distributed machine learning paradigm that enables collaboration among multiple organizations or data silos to train machine learning models while preserving data privacy [ 15 ] . A critical component in an FL system is its communication backend, which manages the exchange of model updates and control messages among participants. The performance of the communication backend has a profound impact on several key facets of the FL process, including convergence speed, communication overhead, fault tolerance, and resource utilization. Fig. 1 : Three common deployment environments for federated learning: (1) LAN: participants in a single building; (2) Geo-Proximal: participants in close geographical proximity; (3) Geo-Distributed: participants in distant locations. Context-aware selection of communication backends is crucial for optimizing performance in each setting. As the demand for FL systems grows and their applications expand, there is an increasing need to understand the performance implications and trade-offs associated with different communication backends, particularly in cross-silo FL scenarios, where options are numerous. This paper aims to provide an in-depth overview and performance comparison of various communication backends employed in cross-silo FL systems, including Message Passing Interface (MPI) [ 8 ] , gRPC [ 10 ] , and PyTorch RPC [ 6 ] . Additionally, in response to the limitations of these communication backends in the context of FL systems, we propose a novel communication backend that combines the capabilities of gRPC and Amazon Simple Storage Service (S3) [ 2 ] . This hybrid approach aims to leverage the strengths of both gRPC and Amazon S3 to deliver enhanced performance, reliability, and scalability, particularly in situations where existing communication backends exhibit shortcomings. We study communication backends for cross-silo FL through point-to-point and end-to-end experiments that vary message size and concurrency across three deployment environments: LAN, Geo-Proximal, and Geo-Distributed. In summary, this work offers the following contributions: 1. Introduction of Hybrid gRPC+S3 backend. A design that routes metadata over RPC and model weights via object storage. It is particularly useful for scenarios that involve transmitting large models across a geo-distributed network. 2. Benchmarking suite across regimes. A comprehensive benchmark suite that includes a variety of models, metrics, and communication backends tailored specifically for cross-silo FL settings. This enables researchers and practitioners to evaluate the performance of different backends across a diverse set of scenarios. 3. Realistic deployment footing. Experiments are conducted in realistic settings that are representative of cross-silo FL environments. We utilize GPU-powered machines and consider various training environments to ensure our results are applicable and insightful to real-world FL tasks. II Background and Motivation II-A Federated Learning Federated learning (FL) trains a shared model across decentralized clients without revealing raw data [ 15 , 20 , 16 ] . There are two primary categorizations of FL: cross-device (many resource-constrained clients, e.g., phones/IoT) and cross-silo (fewer but well-provisioned participants such as hospitals, banks, or data centers). A server coordinates rounds of training by selecting clients, distributing the current global model, collecting locally computed updates (e.g., via SGD [ 21 ] ), and aggregating them. This work focuses on cross-silo FL. II-B Communication Backends for Distributed ML Communication backends move models, gradients, and control messages. Hence, the backend choice strongly affects runtime, scalability, and robustness. We study three widely used options: MPI [ 8 ] (HPC-standard message passing with efficient point-to-point and collectives), gRPC [ 10 ] (HTTP/2-based RPC with Protobuf and support for streaming, and TLS), and PyTorch RPC [ 6 ] (PyTorch framework-integrated RPC supporting sync/async patterns). II-C Motivation (1) FL’s diverse training environments and lack of benchmark studies. Cross-silo FL runs over vastly different networks, from fast intra-site fabrics to high-latency wide area networks (WANs), yet systematic evaluations of communication backends across these regimes are scarce. We quantify this variability on two platforms: a LAN with InfiniBand (up to 5 GB/s bandwidth and 3.17 \mu s latency; TCP fallback about 1 GB/s and 16.8 \mu s) and AWS EC2 with g4dn.2xlarge instances. Table I reports single- and multi-connection bandwidth and latency between North California and seven regions: intra-region links reach \sim 3 GB/s and 0.44 ms, while inter-region links are much slower. The gap between single- and multi-connection throughput widens as latency increases, making connection-level concurrency a first-order factor. This work addresses the benchmarking gap by evaluating backends under LAN, geo-proximal, and geo-distributed conditions with both microbenchmarks and end-to-end training. (2) FL network characteristics and limitations of existing backends. Cross-silo FL deployments require backends that (i) perform under heterogeneous networks with minimal assumptions, (ii) tolerate dynamic participation and failures without global aborts, (iii) scale in both client count and model size while keeping CPU/memory bounded, and (iv) support secure operation over untrusted WAN. Given these needs, we revisit the backends in § II-B for FL. MPI , while efficient, assumes static membership and homogeneous, tightly managed networks. Moreover, SSH/rsh-based process management complicates deployment across independently administered sites, and failure handling often lacks the fault isolation needed when participants can come and go. PyTorch RPC integrates cleanly with the framework but is framework-specific, expects open and stable paths between peers, and offers less control over transport/topology than MPI/gRPC, which can be restrictive under stricter cross-organization security and networking constraints. gRPC is a popular choice for FL applications due to its flexibility and security features. However, it is not optimized for heavy data transfers. For instance, its standard Python implementation multiplexes all traffic over a single HTTP/2 connection. Where multi-connection throughput greatly exceeds single-connection (Table I ), a single channel underutilizes the link. Using many channels can recover throughput for large broadcasts yet raises server memory roughly linearly with concurrency because each send buffers its own copy (Fig. 2 ). Motivated by these constraints, especially for large message transfers across heterogeneous, untrusted networks, we propose gRPC+S3, a communication backend optimized for federated learning scenarios. Region Bandwidth (MB/s) Latency (ms) Single Multi North California 592 2946 0.44 Oregon 133 573 11 North Virginia 39.4 557 32.3 Hong Kong 16.3 513 83.3 Stockholm 11.4 495 90.9 Sao Paulo 8.27 491

Read the complete text