Heterogeneous Federated Learning with Prototype Alignment and Upscaling

Gyuejeong Lee, Jihwan Shin, Daeyoung Choi · 2025

Read Heterogeneous Federated Learning with Prototype Alignment and Upscaling by Gyuejeong Lee, Jihwan Shin, Daeyoung Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04310.txt.

Introduction

Gyuejeong Lee, Jihwan Shin, and Daeyoung Choi present ProtoNorm, a way for many devices to train one model together without sharing raw data. Their paper appeared in 2025. In federated learning, each device trains locally and shares only small summaries. Problems arise when devices hold different kinds of data or run different models. Their method shares one average pattern per class instead of full models.

The team found these class patterns often sit too close together, which blurs decisions. Their fix has two parts. First, the patterns are spread out evenly on a sphere, borrowing an idea from physics. Second, the patterns are scaled up so gaps grow in plain space. Tests on standard data sets beat rival methods. Since only small patterns travel, the method suits phones and other weak devices.

Catalog record

CREATORGyuejeong Lee, Jihwan Shin, Daeyoung Choi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04310

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Heterogeneous Federated Learning with Prototype Alignment and Upscaling Gyuejeong Lee, Jihwan Shin, Daeyoung Choi Abstract Heterogeneity in data distributions and model architectures remains a significant challenge in federated learning (FL). Various heterogeneous FL (HtFL) approaches have recently been proposed to address this challenge. Among them, prototype-based FL (PBFL) has emerged as a practical framework that only shares per-class mean activations from the penultimate layer. However, PBFL approaches often suffer from suboptimal prototype separation, limiting their discriminative power. We propose Prototype Normalization (ProtoNorm), a novel PBFL framework that addresses this limitation through two key components: Prototype Alignment (PA) and Prototype Upscaling (PU). The PA method draws inspiration from the Thomson problem in classical physics, optimizing global prototype configurations on a unit sphere to maximize angular separation; subsequently, the PU method increases prototype magnitudes to enhance separation in Euclidean space. Extensive evaluations on benchmark datasets show that our approach better separates prototypes and thus consistently outperforms existing HtFL approaches. Notably, since ProtoNorm inherits the communication efficiency of PBFL and the PA is performed server-side, it is particularly suitable for resource-constrained environments. 1 Introduction Federated learning (FL) enables collaborative model training across distributed clients without raw data sharing, demonstrating remarkable success in privacy-sensitive domains, including healthcare and finance applications [ 40 ] . As a pioneering FL algorithm, FedAvg [ 26 ] operates by aggregating local model parameters at a central server to create a single global model, which is then distributed back to clients for local training. However, its effectiveness is limited to scenarios with nearly uniform data distributions across homogeneous model architectures. Other traditional FL approaches based on model parameter sharing often face the same limitations. Data and model heterogeneity across clients is prevalent in practice, emerging as a fundamental challenge that demands novel solutions [ 32 , 16 ] . To address these challenges, heterogeneous FL (HtFL) has emerged as a new paradigm that enables clients to employ different model architectures and handle non-uniform data distributions [ 35 ] . One line of approaches adapts parameter sharing by enabling clients to train architecture-specific subnets based on their available resources, addressing both system and statistical challenges [ 7 , 9 , 46 , 2 ] . However, these approaches inherently limit the degree of heterogeneity that can be accommodated, as they require architectural compatibility between shared components. Knowledge distillation (KD) based approaches have been proposed as an alternative that enables an entirely architecture-free federation to overcome this limitation. These approaches focus on knowledge sharing between the server and clients through various representations including logits [ 11 , 15 ] , intermediate features [ 35 , 42 ] , and auxiliary networks [ 30 , 45 ] , offering greater flexibility in handling heterogeneous architectures. Among the KD-based approaches, prototype-based FL (PBFL) has arisen as a particularly effective framework, operating by sharing prototypes—mean activations in the penultimate layer [ 35 ] . This approach offers several practical advantages: higher communication efficiency, enhanced privacy preservation, and implementation simplicity by sharing only feature vectors per class. Despite its advantages, PBFL suffers an inherent constraint in its optimization framework. The separation between global prototypes is solely determined by the weighted averaging of local prototypes, leading to suboptimal inter-prototype discrimination. While recent works like FedTGP [ 42 ] and FedFTP [ 39 ] have addressed this through contrastive learning, the problem of optimal prototype separation remains an open challenge. To address this prototype separability challenge, we propose a novel PBFL framework, Prototype Normalization (ProtoNorm). Our framework consists of two sequentially performed components: Prototype Alignment (PA) and Prototype Upscaling (PU). The PA method draws inspiration from the Thomson problem in classical physics, which seeks minimum energy configurations of N electrons on a unit sphere. By implementing a novel gradient ascent-based Thomson problem solver, PA optimizes global prototype configurations on the unit sphere to maximize angular separation. The server incrementally adjusts each prototype’s position to minimize hyperspherical energy until convergence. Following alignment, the PU method increases prototype magnitudes to enhance separation in Euclidean space—a critical step because training can converge to suboptimal local minima when global prototypes remain unit vectors after PA application. Our motivational experiment on a synthetic spiral dataset visually demonstrates that standard PBFL approaches can fail to separate global prototypes adequately when dealing with non-independent and identically distributed (non-IID) data. In contrast, ProtoNorm achieves better separation of global prototypes even with non-IID data. We further demonstrate that prototype enlargement provides additional prototype separation benefits. Extensive evaluations on popular multi-class benchmark datasets, including Tiny ImageNet, confirm that ProtoNorm consistently outperforms existing PBFL methods and other data-free HtFL approaches. Ablation studies further validate that PA and PU contribute significantly to performance improvements, with their combination delivering superior results compared to either component in isolation. We analyze the convergence of ProtoNorm by examining both the number of PA iterations required across FL rounds and the pairwise global prototype distances during these iterations, confirming stable convergence behavior. Our contributions can be summarized as follows: • We propose ProtoNorm, a novel PBFL framework that significantly improves inter-prototype discrimination through a two-phase approach: (1) PA: optimizing angular separation on the unit sphere, and (2) PU: upscaling prototypes to enhance separation in Euclidean space. • Through experiments on benchmark datasets, we demonstrate that ProtoNorm achieves optimal global prototype separation even with non-IID data, consistently outperforming existing HtFL approaches across various settings. • While maintaining the inherent communication efficiency of PBFL approaches, ProtoNorm performs server-side computation for PA and enhances privacy by eliminating the need for local class distribution information, making it well-suited for resource-constrained FL deployments. 2 Related Work 2.1 Heterogeneous Federated Learning HtFL addresses challenges stemming from heterogeneity in the client model architecture. In this work, we focus on comparing our approach with KD-based HtFL. Existing KD-based HtFL methods can be classified according to the type of information they exchange. Some approaches share model components or parameters. LG-FedAvg [ 17 ] enables clients to share upper model layers while maintaining architecture-specific lower layers. FML [ 30 ] employs mutual distillation [ 43 ] to train and share compact auxiliary models. FeDGen [ 45 ] exchanges a data generator to enhance generalization. Other methods prioritize privacy by avoiding direct parameter sharing. FedDistill [ 11 ] transmits globally averaged class-wise logits from clients to the server. However, this approach risks revealing the number of classes and logit distribution, creating potential privacy vulnerabilities. To address these concerns, FedProto [ 35 ] adopts a more privacy-preserving strategy by exchanging only local prototypes from the penultimate layer. Recent works have further refined PBFL techniques to improve performance. FedTGP [ 42 ] and FedFTP [ 39 ] enhance prototype separability through contrastive learning. Our work also builds upon PBFL and addresses prototype separability but takes a different approach by leveraging geometric insights inspired by the Thomson problem. Moreover, the computation overhead of the PA method scales only with the number of global prototypes, which equals the class count, and remains independent of the number of clients, thus making it particularly suitable for large-scale FL. 2.2 Hyperspherical Approaches in Federated Learning In centralized machine learning, several studies have explored hyperspherical approaches, emphasizing that angular information is important for the semantics of deep networks, particularly in the computer vision

Read the complete text