The More is not the Merrier: Investigating the Effect of Client Size on Federated Learning

Eleanor Wallach, Sage Siler, Jing Deng · 2025

Read The More is not the Merrier: Investigating the Effect of Client Size on Federated Learning by Eleanor Wallach, Sage Siler, Jing Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08198.txt.

Introduction

Eleanor Wallach, Sage Siler, and Jing Deng ask if more clients always help in federated learning. Their twenty twenty-five paper studies a setup where data stays on each client device. Clients train locally and send their models to one hub, which merges them. The best-known merge rule is called FedAvg.

The team finds that as the client count grows, FedAvg accuracy falls by a lot. Their fix is to add a few expert clients who hold large data sets. With these added, accuracy climbs much faster, even with the plain merge rule. They also argue this setup guards privacy well, since raw data stays local.

Catalog record

CREATOREleanor Wallach, Sage Siler, Jing Deng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-08198

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The More is not the Merrier: Investigating the Effect of Client Size on Federated Learning Eleanor Wallach, Sage Siler, Jing Deng Abstract Federated Learning (FL) has been introduced as a way to keep data local to clients while training a shared machine learning model, as clients train on their local data and send trained models to a central aggregator. It is expected that FL will have a huge implication on Mobile Edge Computing, the Internet of Things, and Cross-Silo FL. In this paper, we focus on the widely used FedAvg algorithm to explore the effect of the number of clients in FL. We find a significant deterioration of learning accuracy for FedAvg as the number of clients increases. To address this issue for a general application, we propose a method called Knowledgeable Client Insertion (KCI) that introduces a very small number of knowledgeable clients to the MEC setting. These knowledgeable clients are expected to have accumulated a large set of data samples to help with training. With the help of KCI, the learning accuracy of FL increases much faster even with a normal FedAvg aggregation technique. We expect this approach to be able to provide great privacy protection for clients against security attacks such as model inversion attacks. I Introduction Federated Learning (FL) was coined in 2017 by McMahan et al. as a different approach to machine learning [ 1 ] . As outlined in Fig. 1 , FL occurs when each client trains a machine learning model on their local data (1), sends the trained model’s parameters to a central server (2), a central server aggregates trained models from multiple clients and creates a new global model (3) that is then sent to clients (4) where the process repeats [ 1 ] . Among the many benefits of FL is its ability to protect local data. All training is done locally and none of the data used for training is sent to the global model, giving FL practical use in places such as banks and hospitals [ 2 ] . Privacy reasons like these are one reason why FedAvg was introduced, as it is one of the most prominent advantages of FL [ 1 ] . Since the term’s introduction, there has been intensive research on the topic. Two particular topics that have been looked into in FL are how the distribution of data affects the accuracy, and how the number of clients affects the accuracy. Fig. 1 : An Illustration of Federated Learning Framework. Data can potentially be not identically and independently distributed (non-IID), which presents a problem for federated learning due to the local data of each client not always being representative of the global data. Thus, local updates are not necessarily in line with the global model, resulting in a lower test accuracy when using non-IID data compared to identically and independently distributed (IID) data [ 3 , 4 ] . Previous research has found conflicting information about the effect of the number of clients in FL. Li et al. [ 3 ] found the accuracy of all tested algorithms (FedAvg, FedProx, FedNova, and SCAFFOLD) decreased as the number of clients increased for multiple types of non-IID and IID data. Zhang et al. [ 5 ] found the accuracy of FedAvg, FedProx, and their proposed CSFedAvg increased as the number of clients increased for non-IID data. Wong et al. [ 6 ] observed an increase in the accuracy of FedAvg as the number of clients increased. This improvement was more pronounced for non-IID data compared to IID data. Wu et al. [ 7 ] found FedAvg to achieve a relatively stable accuracy across different numbers of clients with non-IID data. Li et al. [ 8 ] considered 50 clients with a sample ratio of 1 and 100 client with a sample ratio of 0.2. They observed a decrease in accuracy with 100 clients compared to 50 clients for their proposed MOON algorithm and for FedAvg, FedProx, and SCAFFOLD. These papers suggest more research is needed into the effects of different amounts of clients in FL. In this paper, we explore how the number of clients impacts model accuracy. This rest of this paper is structured as follows. In Section II , we review previous research relating to FL, client selection, and privacy. In Section III , we discuss important applications of FL and briefly explain the FedAvg algorithm. In Section IV , we propose a new technique, KCI, to address the loss of accuracy with large numbers of clients. In Section V , we outline our experimental conditions and evaluate our algorithm. We show the superiority of KCI even with large numbers of clients. In Section VI , we summarize our new algorithm and provide future research directions. II Related Work Many papers have worked off of FedAvg previously to propose new algorithms. Li et al. [ 9 ] proposed FedProx and introduced a proximal term to better handle system and data heterogeneity. The proximal term controls the impact of local updates to the global model, achieving model updates more inline with the goals of the global model. Karimireddy et al. [ 10 ] proposed SCAFFOLD which uses cross client variance reduction to correct for "client-drift" by estimating the direction of the global model and each local model. Wang et al. [ 11 ] aggregated the normalized stochastic gradients from clients. Li et al. [ 8 ] proposed MOON and used contrastive learning to improve FL. Clustering has also been proposed as a way to improve FL with non-IID data in particular [ 12 , 13 ] . In clustering frameworks, clients are sorted into groups based on their data distribution, computation capabilities, and other factors. Model aggregation first occurs within each group or cluster. These intermediary model are aggregated by the server to create the global model. Extensive research has looked into how to select clients to participate in each round of FL when data is non-IID and the sample ratio of clients is less than one. Zhang et al. [ 5 ] proposed clients with larger degrees of IID data train the global model more frequently than clients with larger degrees of non-IID data. Wong et al. [ 6 ] considered the varying computational abilities of clients when selecting which clients would participate in training. Xin et al. [ 14 ] proposed FCCPS which can change the number of clients chosen for training based on client performance. Nishio and Yonetani [ 15 ] proposed selecting clients for training based on their computational capacity and data distribution. Data privacy in FL has been greatly researched as privacy is known to be a trade off of higher accuracies [ 16 ] . Privacy is built into the framework of FL since only model parameters are shared, yet there are still privacy concerns. Zhu et al. [ 17 ] showed that local, private data can be recreated from gradients. Since the gradients shared between clients and the server are public, there is risk of local data being exposed. This has led to some algorithms that aim to preserve clients’ privacy while still ensuring high accuracy. Many papers utilize differential privacy (DP), a mathematical method to add noise

Read the complete text