Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers

M Yashwanth [email protected], Sharannya Ghosh [email protected], Aditay Tripathi [email protected], Anirban Chakraborty [email protected] · 2025

Read Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers by M Yashwanth [email protected], Sharannya Ghosh [email protected], Aditay Tripathi [email protected], Anirban Chakraborty [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25372.txt.

Introduction

M Yashwanth, Sharannya Ghosh, Aditay Tripathi, and Anirban Chakraborty propose PEP-FedPT. Their 2025 paper tunes vision transformers in federated settings. Visual prompt tuning adapts pretrained models quickly. It changes few settings for downstream tasks. Federated learning trains across clients without centralizing data. Contact and computation budgets are often tight there.

Global prompt tuning generalizes poorly across heterogeneous clients. Personal tuning can overfit local data instead. Their approach aims for both generalization and personalization. It presents class-contextualized mixed prompts for each input. Class-certain prompts sit alongside a globally shared prompt. Weights come from global class prototypes and priors. That gives per-sample personalization without storing client settings. Prompts are optimized with standard federated averaging. Tests covered CIFAR-100, TinyImageNet, DomainNet, and iNaturalist. Their method always beat leading rivals, they report.

Catalog record

CREATORM Yashwanth [email protected], Sharannya Ghosh [email protected], Aditay Tripathi [email protected], Anirban Chakraborty [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25372

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers M Yashwanth [email protected], Sharannya Ghosh [email protected], Aditay Tripathi [email protected], Anirban Chakraborty [email protected] Abstract Visual Prompt Tuning (VPT) of pre-trained Vision Transformers (ViTs) has proven highly effective for adapting large models to downstream tasks with limited data. Its parameter efficiency makes it particularly suitable for Federated Learning (FL), where both communication and computation are constrained. However, global prompt tuning struggles to generalize across heterogeneous clients, while personalized tuning overfits to local data and lacks generalization. We propose PEP-FedPT (Prompt Estimation from Prototypes for Federated Prompt Tuning), a unified framework designed to achieve both generalization and personalization in federated prompt tuning of ViTs. Within this framework, we introduce the Class-Contextualized Mixed Prompt (CCMP) — based on class-specific prompts maintained alongside a globally shared prompt. For each input, CCMP adaptively combines class-specific prompts using weights derived from global class prototypes and client class priors. This approach enables per-sample prompt personalization without storing client-dependent trainable parameters. The prompts are collaboratively optimized via standard federated averaging. Comprehensive evaluations on CIFAR-100, TinyImageNet, DomainNet, and iNaturalist demonstrate that PEP-FedPT consistently surpasses state-of-the-art baselines under diverse data heterogeneity scenarios, establishing a strong foundation for efficient and generalizable federated prompt tuning of Vision Transformers. 1 Introduction Federated learning (FL) ( McMahan et al. 2017 ) is a collaborative machine learning approach in which a central server coordinates multiple clients to jointly train a global model, while preserving the privacy of the client’s data by keeping the local data decentralized. A major challenge in FL is data heterogeneity: data collected by each client can differ significantly in distribution, resulting in non-identically distributed (non-iid) data across the network. These discrepancies often cause client models to converge to different local minima, a phenomenon known as “client drift” ( Karimireddy et al. 2020 ) , which in turn degrades the generalization performance of the global model. To address this issue, personalized FL methods ( Chen & Chao ; Ma et al. 2022 ; Shamsian et al. 2021 ) have been proposed, aiming to better accommodate diverse local data distributions. However, a key drawback of personalized methods is their reduced generalization performance when applied to new clients ( Deng et al. 2024 ) . Inspired by their success in centralized learning, large foundation models (FMs) are increasingly adopted in FL to mitigate data heterogeneity ( Bommasani et al. 2021 ; Dosovitskiy et al. 2021 ; Radford et al. 2021 ) . FMs demonstrate enhanced robustness in non-iid data settings ( Qu et al. 2022 ) . Yet, their substantial computational demands on resource-constrained edge devices and high communication costs present major hurdles. Parameter-efficient tuning methods, such as Visual Prompt Tuning (VPT) ( Jia et al. 2022 ) , offer a promising solution for efficient FM adaptation within FL, significantly reducing communication overhead while harnessing the power of large models. Prompt tuning has recently gained attention in Federated Learning (FL), with methods like FedPR ( Feng et al. 2023 ) and pFedPG ( Yang et al. 2023 ) exploring its potential. However, each comes with notable limitations. FedPR employs global prompts and is primarily suited for cross-silo FL. This design struggles in highly heterogeneous client settings, where shared global prompts fail to adapt to diverse local data distributions ( Li et al. 2020 ; Deng et al. 2024 ) . pFedPG, on the other hand, generates client-specific prompts at the server and sends them to clients for local fine-tuning. Although personalized, this approach implicitly assumes full client participation in every round—an impractical assumption in real-world FL systems. Moreover, such personalized strategies risk fitting to local data ( Wu et al. ; Deng et al. 2024 ) , limiting their generalization to unseen or non-participating clients (as seen in Table 3 ). To address these issues, SGPT ( Deng et al. 2024 ) employs shared and group-specific prompts to enhance generalization. However, its two-stage training process and non-differentiable mechanism add optimization complexity and computational overhead. Furthermore, when data heterogeneity is high, it still struggles to generalize across diverse client distributions(Tables 1 and 2 ). These limitations highlight a key trade-off in FL prompt tuning: global prompts generalize well but lack expressiveness, while personalized prompts offer local adaptability but suffer from poor generalization and scalability. This raises the central question: Can we achieve effective personalization while relying solely on globally shared prompts? We answer this by proposing a novel prompt-tuning strategy tailored for fine tuning of vision transformer in FL. Our method introduces class-specific prompts that are jointly optimized with shared prompts to address data heterogeneity across clients. To induce personalization without local prompt storage, we propose Prompt Estimation from Prototypes for Federated Prompt Tuning (PEP-FedPT). It generates a Class-Contextualized Mixed Prompt (CCMP) by combining global class-specific prompts. The combination weights are determined by per-class membership scores, computed using global cls -token prototypes and the client’s local class priors. The global cls prototype aggregates class centroids across clients, where each centroid is computed by averaging the cls token representations of data points within a given class. For a given input, we estimate class membership scores—refined by each client’s class priors. These scores act as soft weights to combine class-specific prompts into a single, differentiable mixed prompt (CCMP). This allows each client to dynamically personalize prompts using only shared global information, without the need for local prompt storage or specialized server-side generation. Our approach integrates seamlessly into standard FL pipelines and delivers strong empirical performance across heterogeneous datasets, as demonstrated in Tables 1 and 2 . Despite relying solely on global prompts, it effectively utilizes clients’ data distribution, achieving scalable generalization. Our key contributions are as follows: • We introduce a unified framework PEP-FedPT that jointly optimizes class-specific and shared prompts to address data heterogeneity in federated learning of ViTs. We exploit the clients’ distribution to achieve personalization by using only global prompts. • We design a novel prompt-mixing strategy that generates CCMP as a function of class-specific prompts. We empirically demonstrate its superiority over existing methods through extensive experiments on datasets exhibiting feature and label imbalance. • We provide theoretical insights into our design by showing that CCMP minimizes a quadratic upper bound and is optimal in the Minimum Mean Squared Error (MMSE) sense. 2 Related Work 2.1 Federated Learning (FL): FL is a machine learning paradigm that emphasizes data privacy by enabling collaborative model training under the coordination of a central server, without requiring direct data sharing. FedAvg ( McMahan et al. 2017 ) is the most commonly used technique for aggregating local models. This has led to its broad adoption across various domains, such as Internet of Things and mobile devices ( Mills et al. 2019 ; Nguyen et al. 2021 ; Hard et al. 2018 ; Ramaswamy et al. 2019 ) , healthcare ( Rieke et al. 2020 ; Xu et al. 2021 ; Nguyen et al. 2021 ; Brisimi et al. 2018 ; Feng et al. 2023 ) , person re-identification ( Zhuang et al. 2020 ) , and face recognition ( Liu et al. 2022a ) . Under data-heterogeneity, training the models using FedAvg leads to client-drift . Addressing this, regularization techniques ( Acar et al. ; Li et al. 2020

Read the complete text