FedHPro: Federated Hyper-Prototype Learning via Gradient Matching

Huan Wang, Jun Shen, Haoran Li, Zhenyu Yang, Jun Yan, Ousman Manjang, Yanlong Zhai, Di Wu, Guansong Pang · 2026

Read FedHPro: Federated Hyper-Prototype Learning via Gradient Matching by Huan Wang, Jun Shen, Haoran Li, Zhenyu Yang, Jun Yan, Ousman Manjang, Yanlong Zhai, Di Wu, Guansong Pang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13475.txt.

Introduction

Huan Wang and eight colleagues present FedHPro, federated hyper-prototype learning via gradient matching. It trains across clients while guarding privacy, and prototype variants share global anchors. Current methods update global anchors by averaging or tuning, causing semantic drift and a misaligned global signal. Hyper-prototypes are learnable globals keeping semantic knowledge across clients. Hyper-prototypes are tuned by gradient matching against class traits from clients' real samples. FedHPro boosts separation and uniformity this way, reaching state-of-the-art results on several benchmarks.

Catalog record

CREATORHuan Wang, Jun Shen, Haoran Li, Zhenyu Yang, Jun Yan, Ousman Manjang, Yanlong Zhai, Di Wu, Guansong Pang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94938
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13475

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FedHPro: Federated Hyper-Prototype Learning via Gradient Matching Huan Wang, Jun Shen, Haoran Li, Zhenyu Yang, Jun Yan, Ousman Manjang, Yanlong Zhai, Di Wu, Guansong Pang Abstract Federated Learning (FL) enables collaborative training of distributed clients while protecting privacy. To enhance generalization capability in FL, prototype-based FL is in the spotlight, since shared global prototypes offer semantic anchors for aligning client-specific local prototypes. However, existing methods update global prototypes at the prototype-level via averaging local prototypes or refining global anchors, which often leads to semantic drift across clients and subsequently yields a misaligned global signal. To alleviate this issue, we introduce hyper-prototypes , defined by a set of learnable global class-wise prototypes to preserve underlying semantic knowledge across clients. The hyper-prototypes are optimized via gradient matching to align with class-relevant characteristics distilled directly from clients’ real samples, rather than prototype-level descriptors. We further propose FedHPro , a Federated Hyper-Prototype Learning framework, to leverage hyper-prototypes to promote inter-class separability via mutual-contrastive learning with client-specific margin, while encouraging intra-class uniformity through a consistency penalty. Comprehensive experiments under diverse heterogeneous scenarios confirm that 1) hyper-prototypes produce a more semantically consistent global signal, and 2) FedHPro achieves state-of-the-art performance on several benchmark datasets. Code is available at https://github.com/mala-lab/FedHPro . 1 Introduction Federated Learning (FL) has emerged as a paradigm to collaboratively train a global model across distributed clients based on heterogeneous data, without leaking their privacy ( McMahan et al. 2017 ; Li et al. 2020a ) . However, one major challenge for FL is the data heterogeneity issue ( Li et al. 2022 ) , making the model convergence slow and unstable ( Li et al. 2020d ; Karimireddy et al. 2020 ) . In the practical FL, each client’s data is collected from highly diverse sources according to their own preference space. The non-independent and identically distributed (non-IID) data lead to inconsistency in local objectives among clients ( Xu et al. 2025 ; Li et al. 2020b ) . Meanwhile, the global empirical direction is further distracted by aggregating these skewed local models, resulting in suboptimal performance. Figure 1 : Illustration of heterogeneous FL with domain skew. The Vanilla column visualizes the feature distribution of a standard prototype-based FedProto ( Tan et al. 2022a ) , showing failures in hard domains like SYN. In contrast, the Proposed column shows our approach achieves a larger inter-class distance and a smaller intra-class distance in such domains. Due to its simple aggregation, the prototypes in Vanilla ( ) fail to reliably align with class centers, whereas our Proposed ( ) yields better separability. To mitigate data heterogeneity, a mainstream of subsequent studies focuses on introducing various global signals to facilitate FL training ( Li et al. 2020b ; Mendieta et al. 2022 ; Luo et al. 2021 ) . Considering practicality and decent generalization, prototype-based FL methods ( Tan et al. 2022a ; Yutong et al. 2023 ; Huang et al. 2023 ) have emerged as a novel FL paradigm that transfers global prototypes among clients to tackle data heterogeneity. They average the features with the same class to obtain local prototypes and upload them to the server to produce global prototypes. Afterward, the global prototypes are employed as signals to regularize local updating ( Tan et al. 2022a ) . By aligning with global prototypes, they boost the generalizability ( Zhou et al. 2025 ) . However, existing prototype-based FL methods directly collect the local prototypes from biased data distributions of clients, which unavoidably introduces semantic inconsistency ( Zhou et al. 2023 ; Zhou et al. 2025 ) . This is evident in the feature distribution: FedProto ( Tan et al. 2022a ) shows clear separation between different categories in simple domains ( e.g. , MNIST in Figure 1 ), but exhibits ambiguous clustering in hard domains ( e.g. , SYN in Figure 1 ), particularly around samples near boundaries. Recent prototype-based FL methods, such as FedSA ( Zhou et al. 2025 ) , further improve global prototypes by learning or refining semantic anchors. However, their updates are still primarily performed at the prototype-level by client-specific representations, which may inherit semantic biases and thus fail to provide a consistently aligned global signal. We argue that a common weakness exists in these methods: the class-wise semantic margins between global prototypes could be largely weakened by client-induced representation biases . An intuitive solution is to preserve sufficient local data with diverse distributions to learn a uniform representation space, mirroring centralized training. However, this is fundamentally incompatible with the privacy-preserving FL paradigm. Methods MNIST USPS SVHN SYN AVG \boldsymbol{\uparrow} \triangle FedAvg 96.68 90.43 76.35 51.85 78.82 – \underbrace{\text{FedProto}} 97.40 91.89 79.22 53.15 80.41 +1.59 Using HP 97.48 92.24 80.90 56.39 81.75 +2.93 FedHPro 98.52 93.13 84.95 62.59 84.80 +5.98 Figure 2 : Top : The L_{2} distance of centralized prototypes calculated by centralized training using all clients’ samples to global prototypes from FedAvg ( McMahan et al. 2017 ) and FedProto ( Tan et al. 2022a ) , and from our hyper-prototypes. Bottom : The corresponding accuracy on the Digits ( Peng et al. 2019 ) dataset, ‘ Using HP ’ means replacing the global prototypes with our hyper-prototypes in the FedProto’s loss function. This dilemma inspires our core idea : if the FL model could learn a representation space using all clients’ samples, the obtained prototypes would exhibit better semantic consistency. In other words, we expect to optimize global prototypes over all real samples, instead of adopting the skewed representations from clients. Inspired by this insight, we propose hyper-prototypes , a set of learnable global class-wise prototypes optimized via gradient matching to explicitly align their gradients with the average gradients of real samples from clients. Such gradient alignment, by effectively simulating the optimization trajectories of real samples, enables our hyper-prototypes to distill class-relevant semantic characteristics directly from real samples, rather than prototype-level descriptors. This can be observed by a comparison of prototype generation methods in Figure 2 : hyper-prototypes show a significantly smaller L_{2} distance to centralized prototypes (calculated under centralized training with all clients’ samples) compared to the prototypes from FedAvg ( McMahan et al. 2017 ) , and FedProto ( Tan et al. 2022a ) . The advantages of our hyper-prototypes are further confirmed by the results, where our hyper-prototypes can be used to replace the prototypes in existing methods ( e.g. , FedProto) to achieve largely enhanced performance. Building on hyper-prototypes, we further propose FedHPro , a Federated Hyper-Prototype Learning framework for heterogeneous FL. FedHPro comprises two key components: 1) Hyper-Prototype Contrastive Learning (HPCL), which exploits the hyper-prototypes to construct mutual-contrastive learning ( Chen et al. 2020 ; He et al. 2020 ; Gui et al. 2024 ) with a client-specific margin. HPCL first specifies the margin by measuring pair-wise distances between local prototypes to sharpen decision boundary. By pulling each embedding closer to its assigned hyper-prototypes and pushing it away from others, HPCL promotes inter-class separability while preserving semantics. 2) Hyper-Prototype Alignment Learning (HPAL), where we align each embedding with its representative hyper-prototypes by penalizing deviations at the feature level, thereby improving intra-class uniformity across clients. Our main contributions are listed as: ❶We introduce hyper-prototypes ( Section 2 ), a set of learnable

Read the complete text