Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs

Li Shen, Xiaolei Hao, Qinglun Li, Xiaochun Cao, Zhifeng Hao, Xun Yang · 2026

Read Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs by Li Shen, Xiaolei Hao, Qinglun Li, Xiaochun Cao, Zhifeng Hao, Xun Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10748.txt.

Introduction

Li Shen and co-authors propose FedMITR, a shared model flip and token relabel framework for one-shot shared learning with vision models. In one-shot shared learning a central server learns a global model in a single contact round, but under highly non-IID settings current data-free methods make low-quality data badly misaligned with ground-truth labels. FedMITR trains the global model by fully using all patches of made-up images. It uses sparse model flips during data generation to selectively invert key foregrounds while halting flips of blank backgrounds.

To address meaning-free tokens hindering vision model predictions, patches with high information density use pseudo-labels. Low-density patches are relabelled via group models for robust teaching. Stability analysis shows sparse flips eliminate gradient wobble from background noise, while token relabel reduces gradient spread, together giving a tighter general bound.

Catalog record

CREATORLi Shen, Xiaolei Hao, Qinglun Li, Xiaochun Cao, Zhifeng Hao, Xun Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78371
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10748

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs Li Shen, Xiaolei Hao, Qinglun Li, Xiaochun Cao, Zhifeng Hao, Xun Yang Abstract One-Shot Federated Learning, where a central server learns a global model in a single communication round, has emerged as a promising paradigm. However, under extremely non-IID settings, existing data-free methods often generate low-quality data that suffers from severe semantic misalignment with ground-truth labels. To overcome these issues, we propose a novel Federated Model Inversion and Token Relabel (FedMITR) framework, which trains the global model by fully exploiting all patches of synthetic images. Specifically, FedMITR employs sparse model inversion during data generation, selectively inverting semantic foregrounds while halting the inversion of uninformative backgrounds. To address semantically meaningless tokens that hinder ViT predictions, we implement a differentiated strategy: patches with high information density utilize generated pseudo-labels, while patches with low information density are relabeled via ensemble models for robust distillation. Theoretically, our analysis based on algorithmic stability reveals that Sparse Model Inversion eliminates gradient instability arising from background noise, while Token Relabel effectively reduces gradient variance, collectively guaranteeing a tighter generalization bound. Empirically, extensive experimental results demonstrate that FedMITR substantially outperforms existing baselines under various settings. I Introduction Federated learning (FL) [ 29 ] is a machine learning framework in which multiple clients collaborate to solve machine learning problems under the coordination of a central server or service provider. And the raw data of each client is stored locally and is not exchanged or transferred [ 18 ] . In recent years, FL has shown its potential to facilitate real-world applications in many fields, including recommendation systems [ 23 , 27 ] , medical image analysis [ 26 ] , computer vision [ 28 , 48 ] , and natural language processing [ 7 ] . However, FL poses significant challenges in terms of communication cost and heterogeneity of data between clients. Communication cost is a major bottleneck in FL systems, as clients need to communicate frequently with the server over multiple rounds during the training process. This paradigm presents significant challenges: 1) the heavy communication burden [ 22 , 21 ] , 2) the risk of connection drop errors between clients and the server [ 17 , 5 ] , and 3) the potential risk of man-in-the-middle attacks [ 40 ] and various other privacy or security concerns [ 32 , 43 ] . One-shot FL [ 12 ] has emerged as a solution to these issues by restricting communication rounds to a single iteration, thus mitigating errors arising from multi-round communication and concurrently diminishing the vulnerability to malicious interception. Furthermore, the FL one-shot framework is a particularly contemporary model for market scenarios [ 36 ] where clients predominantly offer pre-trained models. The drawbacks of this method stem from its challenges when dealing with strongly non-iid data [ 1 ] . Due to its single communication session, it is unable to indirectly gather information from other clients through multiple interactions. This leads to a significantly low and unstable accuracy in the single aggregation, as each client’s acquired knowledge is extremely limited. Existing approaches typically employ federated distillation, where a client model ensemble transfers knowledge to the server while simultaneously guiding data synthesis for Data-Free Knowledge Distillation (DFKD) [ 47 , 49 ] . However, our analysis reveals a critical flaw: under severe data heterogeneity, the knowledge embedded in local models diverges significantly. Consequently, standard generators produce low-fidelity data, frequently causing severe semantic misalignment between synthesized data and their assigned labels. To address these challenges, we propose a novel one-shot FL framework named FedMITR, which leverages a ViT-based model inversion framework for server-side data synthesis to train the global model by better utilizing all patches of the generated images. Specifically, we start by synthesizing input images from random noise, without utilizing any additional information from the training data, making it suitable for FL where data privacy is crucial. After a single communication round, we obtain only the model without any data. In a data-free scenario, our approach involves recovering training data from pre-trained client models in some manner and utilizing it for knowledge transfer. Furthermore, due to the dispersed training of client models, the knowledge from each client is limited, resulting in poor quality of synthesized data. Therefore, we need to select sparse patches with different information densities for subsequent processing (Called Sparse Model Inversion). For patches with high information density, they are likely to match pseudo-labels and can be directly used for training the global model. For patches with low information density, we also reuse them by relabeling through ensemble models for knowledge distillation (Called Token Relabel), which ensures that the knowledge contained in the generated data is maximally exploited. Theoretically, we provide a rigorous analysis based on algorithmic stability, proving that FedMITR achieves a strictly tighter generalization bound than traditional dense inversion. Specifically, we demonstrate that Sparse Model Inversion eliminates gradient instability caused by noise, while Token Relabel significantly reduces gradient variance, thereby optimizing the loss landscape with a smaller Lipschitz constant. Empirically, the experimental results indicate that FedMITR significantly improves accuracy compared to existing one-shot FL methods across various heterogeneous data scenarios. For example, FedMITR surpasses the best baselines with 3.20%, 8.92%, and 7.93% on CIFAR10, OfficeHome, and Mini-Imagenet under Dir(0.1) heterogeneous setting, respectively. In summary, our main contributions are summarized as follows: • We pioneer the exploration of Vision Transformers in One-Shot FL and propose a novel framework, FedMITR, to address the limitations of existing DFKD methods. By integrating sparse model inversion with token relabeling, our method effectively utilizes all generated patches to synthesize high-quality pseudo-data. • Our framework is designed exclusively for the server side, requiring no additional training on local clients. This characteristic makes FedMITR highly suitable for contemporary model market scenarios, enabling efficient global model learning without compromising data privacy or requiring extra transmissions. • We provide a rigorous theoretical analysis based on algorithmic stability to validate the generalization superiority of FedMITR. We mathematically prove that our sparse inversion and token relabeling mechanisms stabilize the optimization by reducing the Lipschitz constant of loss gradients, thereby guaranteeing a strictly tighter generalization bound than traditional dense inversion. • Extensive empirical studies on various heterogeneous datasets demonstrate that FedMITR consistently outperforms state-of-the-art baselines. For instance, it achieves accuracy improvements of 3.20%, 8.92%, and 7.93% on CIFAR10, OfficeHome, and Mini-Imagenet, respectively. II Related Work One-Shot Federated Learning. Guha et al. [ 12 ] first propose the concept of One-Shot Federated Learning, which treats local models as an ensemble for final prediction and further introduced the use of knowledge distillation along with public data for this ensemble in a single round of communication. Zhou et al. [ 50 ] refrain from using public data and instead propose transmitting refined local datasets to the server. Li et al. [ 20 ] propose a method utilizing a two-tier knowledge transfer structure FedKT for distillation on public datasets. Instead of using public data, Zhang et al. [ 47 ] propose a data-free method for knowledge distillation by synthesizing data directly from ensemble models on the server-side. Diao et al. [ 8 ] and Heinbaugh et al. [ 15 ] modify the local training phase and by introducing placeholders

Read the complete text