FedHypeVAE: Federated Learning with Hypernetwork-Generated Conditional VAEs for Differentially-Private Embedding Sharing

Sunny Gupta, Amit Sethi · 2026

Read FedHypeVAE: Federated Learning with Hypernetwork-Generated Conditional VAEs for Differentially-Private Embedding Sharing by Sunny Gupta, Amit Sethi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00785.txt.

Introduction

Sunny Gupta and Amit Sethi propose FedHypeVAE in 2026, a way for groups to share data without pooling it raw. In federated work, each site keeps its own data and shares only model updates. The shares here are embeddings, compact codes that stand in for raw items. Prior generators coped poorly when sites' data differed a lot.

Their generator is a variational autoencoder, a model that codes items down and builds them back up. A shared hypernetwork builds a custom decoder for each site, from a private code for that site. It also builds custom starting distributions, sorted by class. Training runs under differential privacy, which adds noise and clips updates before they are shared. Extra terms keep fake codes close to real ones. After training, one neutral code makes samples for new domains. Mixes of codes span many domains at once.

Catalog record

CREATORSunny Gupta, Amit Sethi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00785

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FedHypeVAE: Federated Learning with Hypernetwork-Generated Conditional VAEs for Differentially-Private Embedding Sharing Sunny Gupta, Amit Sethi Abstract Federated data sharing promises utility without centralizing raw data, yet existing embedding-level generators struggle under non-IID client heterogeneity and provide limited formal protection against gradient leakage. We propose FedHypeVAE , a differentially private, hypernetwork-driven framework for synthesizing embedding-level data across decentralized clients. Building on a conditional VAE backbone, we replace the single global decoder and fixed latent prior with client-aware decoders and class-conditional priors generated by a shared hypernetwork from private, trainable client codes. This bi-level design personalizes the generative layerrather than the downstream modelwhile decoupling local data from communicated parameters. The shared hypernetwork is optimized under differential privacy, ensuring that only noise-perturbed, clipped gradients are aggregated across clients. A local MMD alignment between real and synthetic embeddings and a Lipschitz regularizer on hypernetwork outputs further enhance stability and distributional coherence under non-IID conditions. After training, a neutral meta-code enables domain-agnostic synthesis, while mixtures of meta-codes provide controllable multi-domain coverage. FedHypeVAE unifies personalization, privacy, and distribution alignment at the generator level, establishing a principled foundation for privacy-preserving data synthesis in federated settings. Code: github.com/sunnyinAI/FedHypeVAE K eywords Federated Learning \cdot Privacy \cdot Gradient Inversion FedHypeVAE: Federated Learning with Hypernetwork-Generated Conditional VAEs for Differentially-Private Embedding Sharing {}@iitb.ac.in Introduction Deep Neural Networks (DNNs) have driven remarkable progress in medical imaging, yet their widespread clinical deployment remains constrained by limited data availability and stringent privacy requirements litjens2017survey ; shilo2020axes . Medical datasets are often siloed across institutions, while the low prevalence of certain diseases further restricts access to diverse, high-quality training data stacke2020measuring . Although collaborative data sharing could mitigate these challenges, strict regulatory frameworks such as HIPAA and GDPR render centralized dataset aggregation infeasible. To address these limitations, Federated Learning (FL) mcmahan2017communication has emerged as a distributed paradigm that enables multiple institutions to collaboratively train models without exposing raw data. The classical FedAvg algorithm mcmahan2017communication aggregates model updates from clients to construct a global model, ensuring that sensitive data remain within institutional boundaries. However, FL faces several persistent challenges. Communication overhead is substantial—especially with high-capacity architectures such as Vision Transformers (ViTs) dosovitskiy2020image —and performance often degrades under non-IID client distributions. Recent efforts to improve efficiency through lightweight architectures wu2023fediic ; xia2024enhancing have reduced transmission cost but at the expense of robustness and diagnostic fidelity. An emerging alternative is synthetic data sharing , where generative models produce privacy-preserving surrogate datasets instead of transmitting model updates koetzier2024generating ; ktena2024generative . Such methods reduce communication burden and improve cross-domain applicability. While Generative Adversarial Networks (GANs) goodfellow2014generative and diffusion models ho2020denoising achieve high-fidelity synthesis, they remain unstable or computationally expensive for federated environments. In contrast, Variational Autoencoders (VAEs) and their conditional extensions (CVAEs) offer stable, likelihood-based training and computational efficiency, albeit at the cost of reduced perceptual sharpness. Recent work di2024privacy demonstrated that generating data in embedding space rather than image space can preserve task-relevant information while mitigating privacy leakage. This embedding-level paradigm is strengthened by the advent of foundation encoders such as DINOv2 oquab2023dinov2 , which provide compact, semantically rich representations that generalize across imaging domains paul2022vision . Training CVAEs on such embeddings enables the generative model to capture diagnostic features efficiently while reducing redundancy and risk of reconstruction-based attacks. Despite these advances, two fundamental challenges persist. First, existing federated generative frameworks lack the ability to adapt to client-specific heterogeneity, leading to degraded performance under non-IID distributions. Second, formal privacy guarantees are rarely incorporated, with most prior methods relying on heuristic noise injection rather than certified Differential Privacy (DP). Addressing these limitations requires a framework capable of personalized, differentially-private generative modeling that remains consistent and generalizable across diverse clinical domains. To this end, we propose FedHypeVAE —a Federated Hypernetwork-Generated Conditional Variational Autoencoder designed for privacy-preserving, semantically consistent data synthesis across decentralized medical institutions. Unlike prior embedding-based frameworks that rely on a shared global decoder, FedHypeVAE introduces a unified hypernetwork that generates client-specific decoder and class-conditional prior parameters from lightweight private client codes. This design enables client-level personalization while implicitly sharing higher-order generative structure through the hypernetwork, thereby improving adaptability under non-IID conditions. Each client trains a local conditional VAE on embeddings extracted from a frozen foundation model (e.g., DINOv2), while the shared hypernetwork parameters are optimized collaboratively via Differentially Private Stochastic Gradient Descent (DP-SGD), ensuring formal (\varepsilon,\delta) -privacy against gradient inversion and membership inference attacks. Furthermore, a Maximum Mean Discrepancy (MMD) -based alignment regularizer enforces cross-site distributional coherence, and a meta-code synthesis module learns a domain-agnostic latent code for globally representative embedding generation. Contributions. Our main contributions are threefold: • We introduce FedHypeVAE , the first federated framework that integrates hypernetwork-based parameter generation with conditional VAEs to enable privacy-preserving embedding synthesis. • We formulate a principled bi-level federated optimization strategy that jointly learns personalized client decoders and a globally consistent hypernetwork under certified (\varepsilon,\delta) -DP guarantees via gradient clipping and calibrated Gaussian noise. • We propose an MMD-based alignment and meta-code generation mechanism that ensure cross-domain coherence and high-fidelity synthetic embedding generation with minimal privacy–utility trade-off. Extensive multi-institutional experiments on diverse medical imaging datasets demonstrate that FedHypeVAE substantially outperforms existing federated generative baselines in terms of robustness, generalization, and privacy compliance. By combining foundation model embeddings, hypernetwork-driven personalization, and differential privacy, FedHypeVAE establishes a new paradigm for secure and effective data sharing in federated medical AI. Related Work Gradient inversion and privacy in federated learning Federated learning (FL) reduces the need for centralized data aggregation by training models through decentralized gradient exchanges across clients. However, a substantial body of research on gradient inversion and reconstruction attacks has demonstrated that shared updates (gradients or parameter deltas) can leak sensitive information, including approximate input reconstructions, membership inference, and attribute disclosure ( fredrikson2015model ; zhu2019deep ; geiping2020inverting ) . These risks are amplified in regimes involving highcapacity vision encoders and heterogeneous, small-scale medical datasets, where local gradients become more tightly coupled to individual training samples. This vulnerability motivates defenses that either (i) minimize the exposure surface by communicating compressed or less informative representations, or (ii) alter the communication primitive so that only aggregated or masked informationrather than raw updatesis revealed to the central server. Privacy-preserving techniques in federated learning Privacy-preserving FL methods primarily fall into three methodological categories. (1) Secure multi-party computation (SMC) and secure aggregation conceal individual updates by allowing the server to observe only aggregatedresults, thereby preventing direct reconstruction of any client’s gradients ( yao1982protocols ; bonawitz2017practical ; mugunthan2019smpai ; mou2021verifiable ) . (2) Homomorphic encryption (HE) enables mathematical operations to be performed directly on encrypted parameters, but typically introduces prohibitive computational and communication overhead ( gentry2009fully ; park2022privacy ; ma2022privacy ) . (3) Differential privacy (DP) enforces formal privacy guarantees by clipping and perturbing updates with calibrated noise ( geyer2017differentially ; mcmahan2017learning ; yu2020salvaging ; bietti2022personalization ; shen2023share ) . In addition, empirical defensessuch as gradient pruning, masking, or stochastic noise injection ( zhu2019deep ; huang2021evaluating ; li2022auditing ; wei2020framework ) as well as specialized systems like Soteria , PRECODE , and FedKL ( sun2020provable ; scheliga2022precode ; ren2023gradient ) have been proposed to mitigate leakage. Nonetheless, these techniques often struggle with a persistent privacyutility trade-off , where stronger protection degrades model accuracy and cross-domain generalization. Such limitations motivate more

Read the complete text