OvA-LP: A Simple and Efficient Framework for Federated Learning on Non-IID Data
Dongjin Park, Hasung Yeo, Joon-Woo Lee · 2025
Read OvA-LP: A Simple and Efficient Framework for Federated Learning on Non-IID Data by Dongjin Park, Hasung Yeo, Joon-Woo Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05028.txt.
Introduction
Dongjin Park, Hasung Yeo, and Joon-Woo Lee present OvA-LP, a light way to fine-tune base models on spread-out data whose mix differs by site. Local drift, where each site pulls the shared model its own way, is the main threat in such federated work.
OvA-LP fights drift at its source. It pairs a frozen encoder with simple probing and a one-vs-all head, in a two-stage run that keeps learned features intact. On CIFAR-100 with one hundred clients, it held most of its even-mix score, while strong rivals kept only a small share of theirs.
Catalog record
| CREATOR | Dongjin Park, Hasung Yeo, Joon-Woo Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-05028
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OvA-LP: A Simple and Efficient Framework for Federated Learning on Non-IID Data Dongjin Park, Hasung Yeo, Joon-Woo Lee Abstract Federated fine-tuning (FFT) adapts foundation models to decentralized data but remains fragile under heterogeneous client distributions due to local drift, i.e., client-level update divergences that induce systematic bias and amplified variance in the global model. Existing aggregation and personalization methods largely correct drift post hoc, which proves brittle under extreme non-IID conditions. We introduce OvA-LP, a minimalist framework that is, to our knowledge, the first explicitly designed to suppress drift at its source within the PEFT-based FFT paradigm. OvA-LP combines linear probing on a frozen encoder with a one-vs-all head and a simple two-stage procedure, preserving pretrained feature geometry and decoupling logits to prevent the mechanisms that amplify drift. On CIFAR-100 with 100 clients, averaged over shard-1, shard-2, and Bernoulli–Dirichlet partitions, OvA-LP retains 95.9% of its IID accuracy, whereas state-of-the-art FFT baselines retain only 10.1% (PFPT) and 34.5% (FFT-MoE) under the same conditions. OvA-LP further maintains resilience under both symmetric and asymmetric label noise. In addition, precomputing encoder features makes per-round cost nearly independent of encoder size. Together, these results demonstrate that OvA-LP provides a principled and efficient basis for robust FFT under heterogeneity. 1 Introduction Foundation models (FMs) have reshaped machine learning by providing powerful pretrained representations that can be adapted to diverse downstream tasks. In federated learning (FL), this shift has given rise to federated fine-tuning (FFT), where clients adapt a shared encoder instead of training models from scratch [ 28 ] . Parameter-efficient fine-tuning (PEFT) methods such as adapters, LoRA, and prompt tuning [ 6 , 7 , 10 ] further reduce computational and communication costs, making FFT a practical paradigm for large-scale decentralized adaptation. This paradigm is particularly critical in domains where data locality is a strict requirement, such as training on sensitive medical records across hospitals or financial data across institutions. However, despite its efficiency and practical importance, the robustness of FFT under heterogeneous client distributions remains a major challenge [ 15 ] . This gap motivates a rethinking of how to make FFT robust to extreme heterogeneity. The core difficulty lies in local drift: client updates diverge due to distributional differences, and when aggregated, these drifts bias and destabilize the global model. Specifically, client-level divergences manifest as both systematic bias and amplified variance in the aggregated update, degrading final accuracy and hindering convergence. In practice, under extreme non-IID conditions, state-of-the-art FFT methods often converge slowly and retain well below half of their IID accuracy within 50 rounds. This persistent relative gap highlights the need for approaches that directly mitigate drift at the client level, rather than merely compensating for it after aggregation. Broadly, prior efforts fall into two families. Aggregation strategies modify the global update rule, from classical methods such as FedProx and Scaffold [ 11 , 9 ] to more recent LoRA-specific variants [ 19 , 5 , 22 ] . Personalization frameworks attach client-specific modules to absorb drift locally, ranging from adapters and prompts to expert-based models [ 25 , 4 , 17 , 23 ] . In practice these families are not disjoint—many approaches combine both, such as FFT-MoE [ 8 ] with expert routing on top of FedAvg, or PFPT [ 20 ] with redesigned prompts and aggregation. Yet despite their variety, they share a common philosophy: drift is treated as unavoidable and corrected only post-hoc, once it has already manifested at the client or global level. This reactive stance leaves them fragile under extreme heterogeneity, as no method so far has succeeded in preventing drift from arising in the first place. In this paper, we present OvA-LP, a minimalist framework that suppresses client drift at its root. OvA-LP integrates three lightweight components—linear probing (LP) on a frozen encoder [ 1 ] , one-vs-all (OvA) binary heads, and a two-stage training schedule—each explored in isolation but never unified. By aligning them within a bias–variance decomposition of federated gradients, we systematically connect feature geometry, label decoupling, and variance control into a single source-level framework. This reframes drift mitigation from post-hoc correction to proactive prevention, suggesting shift in how robustness is pursued within PEFT-based FFT. Our main findings are: • OvA-LP consistently prevents local drift from arising at the client level, offering a principled foundation for robust FFT under heterogeneity. • OvA-LP retains 95.9% of its IID accuracy on CIFAR-100 with 100 clients under shard-1, shard-2, and Bernoulli–Dirichlet ( p=0.1,\alpha=0.001 ) [ 21 ] , whereas FFT-MoE and PFPT retain only 34.5% and 10.1%, respectively. • OvA-LP demonstrates innate robustness to label noise: it consistently reduces accuracy degradation under both symmetric and asymmetric corruption, maintaining resilience at higher noise levels and surpassing specialized noise-robust baselines. • OvA-LP precomputes encoder features once, making per-round training nearly independent of encoder size and preserving modularity for integration with other FFT strategies. 2 Related Work Aggregation strategies. A long line of work has sought to improve FL robustness by modifying the global update rule. Classical approaches such as FedProx and Scaffold reduce the variance of client updates and partially stabilize convergence. More recent extensions adapt these ideas to PEFT settings, for example FLoRA, FedSA-LoRA, and FRLoRA [ 19 , 5 , 22 ] . While effective in mitigating some client drift, these methods still rely on aggregation at the server side, typically applied only after local divergence has already occurred. Personalization frameworks. Another direction attaches client-specific modules to absorb drift locally. Examples include FedAdapter and FedPrompt [ 2 , 26 ] , as well as expert-based extensions such as FFT-MoE and PFPT. These approaches improve local adaptation, but global consistency remains limited because personalization cannot prevent drift from propagating into the shared model. Classification heads for label imbalance. Another line of work modifies the classification head to mitigate skewed label distributions. FedRS [ 12 ] restricts softmax updates for missing classes, mitigating bias under label imbalance. OvA-based approaches such as FedOVA, FedABC, and ATHENA-FL [ 27 , 16 , 3 ] decompose multiclass tasks into binary classifiers to avoid softmax coupling and improve fairness. However, these methods are designed for scratch training and focus mainly on label imbalance, without addressing the broader challenge of feature drift. Label noise robustness. A complementary line of research tackles noisy labels in FL. Methods such as FedCorr [ 21 ] and FedLTF [ 24 ] design correction mechanisms or robust objectives to improve performance under corruption. While effective, these approaches explicitly address noise rather than the underlying drift mechanisms, and remain orthogonal to our focus. Our positioning. Unlike prior OvA-based methods restricted to scratch training and label imbalance, OvA-LP is designed to prevent drift from arising by freezing the encoder and introducing a two-stage OvA head. Its minimalist design remains modular and in principle compatible with aggregation and personalization families, suggesting potential for deployment across diverse FFT pipelines. 3 Methodology Figure 1: Overall structure of OvA-LP. Clients precompute encoder features once (left) and perform two-stage local training with one-vs-all heads (right). Overview. OvA-LP is motivated by a source-level philosophy: preventing drift at its origin rather than correcting it post hoc. Fig. 1 summarizes the overall workflow. Clients first precompute encoder features with a frozen backbone, then train one-vs-all heads under a