UniVarFL: Uniformity and Variance Regularized Federated Learning for Heterogeneous Data
Sunny Gupta, Nikita Jangid, Amit Sethi · 2025
Read UniVarFL: Uniformity and Variance Regularized Federated Learning for Heterogeneous Data by Sunny Gupta, Nikita Jangid, Amit Sethi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08167.txt.
Introduction
Sunny Gupta, Nikita Jangid, and Amit Sethi present a federated learning scheme that keeps local training honest when each site's data look different. Their paper appeared in two thousand twenty-five. Federated learning trains one model across many sites without pooling data. Skewed local data bias each site's classifier and drag the shared model down.
Their method, UniVarFL, fixes the bias at the client, with no help from the global model. One penalty aligns each class's predicted odds with what balanced data would give. Another spreads feature points evenly over their sphere, which aids generalization. Together they mimic training on pooled, balanced data. Tests across several benches beat existing methods in accuracy. The scheme stays light, suiting phones and other weak devices. Code is shared in public.
Catalog record
| CREATOR | Sunny Gupta, Nikita Jangid, Amit Sethi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08167
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UniVarFL: Uniformity and Variance Regularized Federated Learning for Heterogeneous Data Thanks: Citation : Authors. Title. Pages…. DOI:000000/11111. Sunny Gupta, Nikita Jangid, Amit Sethi Abstract Federated Learning (FL) often suffers from severe performance degradation when faced with non-IID data, largely due to local classifier bias. Traditional remedies such as global model regularization or layer freezing either incur high computational costs or struggle to adapt to feature shifts. In this work, we propose UniVarFL , a novel FL framework that emulates IID-like training dynamics directly at the client level, eliminating the need for global model dependency. UniVarFL leverages two complementary regularization strategies during local training: Classifier Variance Regularization , which aligns class-wise probability distributions with those expected under IID conditions, effectively mitigating local classifier bias; and Hyperspherical Uniformity Regularization , which encourages a uniform distribution of feature representations across the hypersphere, thereby enhancing the model’s ability to generalize under diverse data distributions. Extensive experiments on multiple benchmark datasets demonstrate that UniVarFL outperforms existing methods in accuracy, highlighting its potential as a highly scalable and efficient solution for real-world FL deployments, especially in resource-constrained settings. Code: https://github.com/sunnyinAI/UniVarFL 1 Introduction Federated Learning (FL) [ 1 ] enables the training of deep neural networks across decentralized devices, eliminating the need for central data collection while often achieving performance comparable to centralized methods [ 2 ] . However, because each device typically holds non-independent and non-identically distributed (non-IID) data, local training may converge to suboptimal local optima—a phenomenon known as client drift —which diverges from the global objective. To mitigate client drift, several studies have regularized local models using the global model as a reference [ 3 , 4 , 5 ] . More recent work has shown that the classifier (i.e., the final layer) is particularly prone to bias in non-IID settings [ 6 ] . Approaches such as using augmentation techniques [ 7 ] or randomly initializing and freezing classifier weights to enforce orthogonality [ 8 , 9 , 10 ] have been proposed. However, these methods generally reduce bias indirectly and often incur significant computational overhead—state-of-the-art methods like FedProx [ 5 ] , SCAFFOLD [ 11 ] , and FedDyn [ 3 ] require layer-wise weight comparisons, while MOON [ 4 ] and FedAlign [ 12 ] demand additional forward passes. Moreover, many of these techniques are evaluated primarily in label-shift settings (often simulated with the Dirichlet distribution) and may not effectively address feature shifts. To address these issues, we introduce UniVarFL , a novel framework that directly emulates IID-like training dynamics through two complementary regularization strategies: 1.1 Classifier Variance Regularization This strategy harmonizes class-wise probability variances across clients to counteract local bias. By ensuring that the classifier’s output does not disproportionately favor a subset of classes—an effect commonly observed in non-IID settings [ 6 ] . UniVarFL promotes balanced predictions. This regularization minimizes the divergence between locally optimized classifiers and the ideal IID condition, thereby reducing client drift without relying on computationally expensive global comparisons. 1.2 Hyperspherical Uniformity UniVarFL also enforces a uniform distribution of feature representations on a hypersphere. This regularization discourages the encoder from overfitting to local, biased features by penalizing concentrated representations. As a result, the learned features are more evenly spread out, enhancing generalization across diverse data distributions. This approach is inspired by observations in which constrained classifier weight dynamics (e.g., via singular value decomposition) reveal degraded variance in non-IID settings [ 8 , 9 , 10 ] . Together, these strategies enable local models to bridge the gap between IID and non-IID training conditions. Unlike prior methods that indirectly reduce bias through global regularization, UniVarFL directly promotes IID-like dynamics at the local level. Extensive evaluations on diverse benchmarks—spanning both label-shift and feature-shift non-IID scenarios—demonstrate that UniVarFL achieves state-of-the-art performance. Notably, it delivers superior accuracy, faster convergence, and enhanced computational efficiency compared to existing methods such as FedProx [ 5 ] , SCAFFOLD [ 11 ] , FedDyn [ 3 ] , MOON [ 4 ] , and FedAlign [ 12 ] . 2 Related Work 2.1 Federated Learning (FL) Federated Learning (FL) is a distributed approach that supports privacy-preserving model training by ensuring raw data remains on local devices. The canonical workflow, as illustrated by FedAvg [ 1 ] , involves four main steps. Initially, a central server broadcasts the current global model to all participating clients. Each client then refines this model using its locally stored data and sends the updated parameters back to the server. Finally, the server aggregates these local updates into a new global model. While FL encompasses a broad range of research directions—including client selection [ 13 , 14 ] , data sharing [ 15 ] , privacy [ 16 , 17 ] , communication efficiency [ 18 , 19 , 20 ] , medical applications [ 21 ] , and knowledge distillation [ 22 , 23 ] —our work focuses on mitigating performance degradation arising from non-IID data distributions. These efforts typically involve adjustments to either the aggregation procedure or the local training process. 2.2 Aggregation Regularization Aggregation-based methods maintain the standard local training routine while modifying how client models are aggregated. For instance, FedAvgM [ 24 ] incorporates momentum during aggregation, while FedNova [ 20 ] normalizes local updates before averaging. In personalized FL, hypernetworks have been employed to generate client-specific parameters [ 25 , 26 ] . Other techniques—such as PFNM [ 27 ] and FedMA [ 28 ] leverage Bayesian methods to align neurons across clients, while PAN [ 29 ] integrates positional encoding for enhanced efficiency. Additionally, FedDF [ 30 ] utilizes generated unlabeled data, and CCVR [ 6 ] addresses the classifier’s bias (typically the most vulnerable layer) by employing a Gaussian mixture model to calibrate artificial data. 2.3 Local Training Regularization Local training regularization strategies adjust the client-side optimization without altering the aggregation step. Methods like FedProx [ 5 ] add an L2 penalty between local and global model weights to limit divergence, while FedDyn [ 3 ] introduces an inner product term between consecutive local models to stabilize convergence. MOON [ 4 ] further refines this approach by incorporating a contrastive loss that balances representational similarities among the global model, the current local model, and the previous iteration. Although these techniques leverage the global model as a guiding reference, they encounter two main limitations. First, the global model—being an average of local updates—can inherit bias from its constituent models. Second, the additional computations (such as L2 distance measurements and multiple forward passes) can hinder scalability. Recent advancements demonstrate that effective regularization need not depend on the global model. For example, FedAlign [ 12 ] employs GradAug [ 7 ] for self-regularization, albeit with extra computational overhead, while FedBABU [ 8 ] reduces classifier bias by randomly initializing and freezing the classifier weights. Inspired by these insights, our proposed framework, UniVarFL , operates solely at the local training level. UniVarFL directly emulates IID-like training dynamics by integrating two novel regularization strategies— Classifier Variance Regularization and Hyperspherical Uniformity —to mitigate non-IID degradation without incurring heavy computational costs. Figure 1: Presents the singular values of the classifier’s final layer weights, obtained from a model trained on the CIFAR-100 dataset. Algorithm 1 UniVarFL: Federated Learning with Variance and