TinyProto: Communication-Efficient Federated Learning with Sparse Prototypes in Resource-Constrained Environments

Gyuejeong Lee, Daeyoung Choi · 2025

Read TinyProto: Communication-Efficient Federated Learning with Sparse Prototypes in Resource-Constrained Environments by Gyuejeong Lee, Daeyoung Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04327.txt.

Introduction

Gyuejeong Lee and Daeyoung Choi wrote a 2025 paper on training shared computer models without pooling private data. In federated learning, many devices train a joint model by sending updates, not raw data. On weak devices with slow links, those updates are still too big, since the usual method sends a summary for every class.

Their method, TinyProto, thins those summaries. Each class gets its own slice of the summary, and only the filled parts are sent. Sizes also adjust to how common each class is on a device. Tests cut the data sent by up to four times with no loss in accuracy. The approach adds no work on the device itself and lets unlike devices join the same training.

Catalog record

CREATORGyuejeong Lee, Daeyoung Choi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04327

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TinyProto: Communication-Efficient Federated Learning with Sparse Prototypes in Resource-Constrained Environments Gyuejeong Lee, Daeyoung Choi Abstract Communication efficiency in federated learning (FL) remains a critical challenge for resource-constrained environments. While prototype-based FL reduces communication overhead by sharing class prototypes—mean activations in the penultimate layer—instead of model parameters, its efficiency decreases with larger feature dimensions and class counts. We propose TinyProto, which addresses these limitations through Class-wise Prototype Sparsification (CPS) and adaptive prototype scaling. CPS enables structured sparsity by allocating specific dimensions to class prototypes and transmitting only non-zero elements, while adaptive scaling adjusts prototypes based on class distributions. Our experiments show TinyProto reduces communication costs by up to 4× compared to existing methods while maintaining performance. Beyond its communication efficiency, TinyProto offers crucial advantages: achieving compression without client-side computational overhead and supporting heterogeneous architectures, making it ideal for resource-constrained heterogeneous FL. 1 Introduction Federated learning (FL) has emerged as a transformative framework for collaborative model training without the need for centralized data aggregation ( McMahan et al. 2017 ) . It offers substantial advantages, including enhanced data privacy, optimized computational resource utilization, and improved communication efficiency ( Zhang et al. 2021a ) . Among these benefits, reducing communication overhead is particularly critical for the scalability of FL ( Sattler et al. 2019 ) . Traditional FL frameworks transmit model weights or gradients, rather than raw data, between the central server and clients, thereby conserving communication resources. However, as deep networks continue to grow in size, the communication overhead associated with transmitting model parameters increases significantly, undermining these advantages. This presents substantial challenges for large-scale deployments in resource-constrained environments, such as Internet of Things (IoT) networks ( Nguyen et al. 2021 ) . To address this challenge, various strategies have been proposed to optimize either communication frequency ( Karimireddy et al. 2020 ; Wang and Joshi 2019 ; Wang et al. 2019 ) or communication volume, with this work focusing specifically on the latter—reducing communication volume. Some approaches achieve this by transmitting only a subset of deep network parameters ( Chen and Chao 2021 ; Wang et al. 2024 ) or gradients ( Sattler et al. 2019 ; Han et al. 2020 ) , while others utilize low-rank representations of parameter matrices ( Wang et al. 2023 ; Wu et al. 2024 ; Wu et al. 2022 ) . Additionally, techniques such as pruning model weights have proven effective in mitigating communication overhead ( Jiang et al. 2022 ; Zhang et al. 2022 ; Wu et al. 2023 ) . In contrast to transmitting model parameters, alternative methods exchange knowledge between the server and clients by sharing logits ( Jeong et al. 2018 ; Li and Wang 2019 ) , intermediate features ( Tan et al. 2022 ; Zhang et al. 2024 ) , auxiliary networks ( Shen et al. 2020 ) , or data generators ( Zhu et al. 2021 ) . Although these methods were initially developed to address challenges such as data and model heterogeneity, they also contribute to improving communication efficiency. Figure 1 : Comparison of communication costs between FedAvg and PBFL approaches for ShuffleNet v2, showing parameter counts across different numbers of classes and prototype dimensions. Among these approaches, prototype-based FL (PBFL) stands out for its communication costs being less sensitive to model size and its straightforward implementation, while providing natural privacy protection. PBFL’s core mechanism involves sharing prototypes—represented as mean activations in the penultimate layer for each class—instead model parameters. While PBFL offers notable advantages in communication efficiency, our analysis reveals significant scalability limitations. Consider a classification task using ShuffleNet v2 (approximately 1.36M parameters for 1000 classes, with 1.0M parameters between the penultimate and output layers) as illustrated in Figure 1 . PBFL’s communication costs, determined by the product of class count and prototype dimensions, are significantly lower than FedAvg ( McMahan et al. 2017 ) when these factors are small. However, this advantage diminishes rapidly as either the number of classes or prototype dimensions increases, limiting PBFL’s effectiveness in resource-constrained networks. In this work, we address the challenge of mitigating the communication overhead caused by scalability issues in the PBFL approach, aiming to make it more practical for resource-constrained environments. We propose TinyProto, a novel framework that combines prototype sparsification and prototype scaling. The Class-wise Prototype Sparsification (CPS) method reduces communication overhead by introducing structured sparsity, where each class prototype uses only specific dimensions while zeroing out the others. Beyond reducing communication costs, CPS improves classification performance by creating more discriminative prototypes through structured sparsification. While PBFL methods like FedProto and FedTGP show promise for heterogeneous FL (HtFL), they often underperform compared to other HtFL approaches, making the direct application of CPS impractical. To address this limitation, we introduce a prototype scaling method that adjusts prototypes based on class-wise sample sizes. By integrating CPS with prototype scaling, TinyProto achieves substantial communication efficiency without compromising model performance. Experimental results demonstrate that TinyProto reduces communication costs by up to 10× compared to the original PBFL approaches and 4× compared to the most efficient baselines, all while delivering superior performance. In summary, this work addresses a key challenge in PBFL: reducing communication overhead while maintaining model performance . We make three key contributions: • We introduce CPS and adaptive prototype scaling—a novel method that achieves structured sparsity while enhancing classification through discriminative prototypes and class-wise adjustments. • We propose the TinyProto framework, integrating these methods to achieve up to 10× communication cost reduction while enhancing performance. • TinyProto distinguishes itself by eliminating client-side computation and fine-tuning requirements for model pruning while supporting heterogeneous architectures, making it particularly effective for resource-constrained, real-world deployments. 2 Related Work 2.1 Heterogeneous Federated Learning HtFL addresses challenges arising from heterogeneity in data and client model architectures. Existing HtFL approaches can be broadly categorized based on the type of information exchanged: model parameters, auxiliary model or data generator parameters, and logits or intermediate features. To accommodate varying model architectures, LG-FedAvg ( Liang et al. 2020 ) allows clients to share upper model layers while retaining architecture-specific lower layers. Auxiliary model-based methods, such as FML ( Shen et al. 2020 ) , use mutual distillation ( Zhang et al. 2018 ) to train and share compact models. Similarly, sharing a data generator improves generalization ( Zhu et al. 2021 ) . Other approaches avoid sharing model parameters by using logits or intermediate features. FedMD ( Li and Wang 2019 ) , for instance, uses a global public dataset to enable knowledge transfer among heterogeneous clients. FedDistill ( Jeong et al. 2018 ) transmits globally averaged class-wise logits from clients to the server, but this method risks exposing the number of classes and the logit distribution, leading to potential privacy concerns. To address these risks, FedProto ( Tan et al. 2022 ) takes a privacy-preserving approach by exchanging local prototypes of the penultimate layer. Building on these advancements, recent works have refined PBFL techniques further. FedTGP ( Zhang et al. 2024 ) enhances performance by employing adaptive-margin-enhanced contrastive learning (ACL), which improves prototype separability. Similarly, our work builds upon PBFL but focuses on enhancing communication efficiency by introducing sparsity in prototypes. 2.2 Sparsity in Federated Learning Connection pruning ( Han et al. 2015a ; Han et

Read the complete text