Loss Gap Parity for Fairness in Heterogeneous Federated Learning

2026

Read Loss Gap Parity for Fairness in Heterogeneous Federated Learning (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29818.txt.

Introduction

Erraji, Perrot, and Bellet target fairness in federated learning with unlike clients. Clients join to do better on rare local data, yet still judge the shared model at home. The team defines fairness through the loss gap: global loss minus best local loss. Their EAGLE method shrinks gaps between clients. It favors clients farthest from their local best. They prove convergence for non-convex losses. Tests keep utility competitive with strong baselines.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85313
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.29818

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Loss Gap Parity for Fairness in Heterogeneous Federated Learning Abstract While clients may join federated learning to improve performance on data they rarely observe locally, they often remain self-interested, expecting the global model to perform well on their own data. This motivates an objective that ensures all clients achieve a similar loss gap —the difference in performance between the global model and the best model they could train using only their local data. To this end, we propose EAGLE, a novel federated learning algorithm that explicitly regularizes the global model to minimize disparities in loss gaps across clients. Our approach is particularly effective in heterogeneous settings, where the optimal local models of the clients may be misaligned. Unlike existing methods that encourage loss parity, potentially degrading performance for many clients, EAGLE targets fairness in relative improvements. We provide theoretical convergence guarantees for EAGLE under non-convex loss functions, and characterize how its iterates perform relative to the standard federated learning objective using a novel heterogeneity measure. Empirically, we demonstrate that EAGLE reduces the disparity in loss gaps among clients by prioritizing those furthest from their local optimal loss, while maintaining competitive utility in both convex and non-convex cases compared to strong baselines. Loss Gap Parity for Fairness in Heterogeneous Federated Learning Brahim Erraji Michaël Perrot Aurélien Bellet Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000 Lille, France Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStALF-59000 Lille, France PreMeDICaL team, Inria, Idesp, Inserm, Université de Montpellier 1 INTRODUCTION Federated learning enables the training of machine learning models based on data stored across multiple clients without accessing or sharing it directly ( 13 ) . It has emerged as a promising training framework for applications with regulated data access such as healthcare ( 11 ) , banking ( 19 ) and other sensitive applications ( 8 ) . Federated learning is typically orchestrated by a centralized, often trusted server responsible for aggregating local client models and synchronizing updates. This is the setting we consider in our work. Federated optimization algorithms typically aim to minimize a (weighted) average of the local client training losses by generating a sequence of models based on information received from the clients, such as the gradient of the loss function on their respective data. Clients participate in federated learning by contributing their resources in exchange for access to knowledge aggregated from other clients, enabling the training of a model that generalizes well to the overall data distribution. However, they often also expect the model to perform well on their own local data distribution. For example, consider a medical diagnosis system in which each hospital seeks a model that generalizes to rare or unseen patient cases, while still maintaining high performance on the types of patients commonly encountered in its region. In this context, a model that exhibits uneven performance across datasets from different hospitals can be viewed as unfair. In fact, Federated Averaging (FedAvg)—introduced by 13 and now the default algorithm in federated learning—often rewards clients that possess larger datasets, regardless of the relevance of their data to the overall utility. This happens because these clients are (i) typically weighted by their data size during aggregation; and (ii) allowed to run more local steps, since with a fixed batch size larger datasets require more updates per epoch ( 23 ) , steering the global optimization toward their minimizers. These factors raise concerns about unfairness toward other clients, potentially discouraging their participation. As a result, promoting fairness has become a fundamental requirement in federated learning. Similar to algorithmic fairness in centralized machine learning, determining what makes a federated model fair remains subject to debate and often depends on the specific application ( 21 ; 1 ; 20 ) . A widely studied notion of fairness in federated learning is loss parity ( 10 ; 9 ; 25 ) , which aims to equalize local losses between all participating clients. The underlying intuition is that similar loss values indicate the model has learned each client’s data distribution equally well. However, this implicitly assumes that losses are directly comparable across clients—a condition that often fails in heterogeneous settings. In practice, the data distributions of some clients may be intrinsically more challenging to model or suffer from lower quality or resolution, causing their losses to be on inherently different scales. Enforcing equalized losses in such cases can distort optimization and reduce the utility of the global model, sometimes causing a leveling down effect—where fairness degrades outcomes for the best-off groups without benefiting the worst-off ( 15 ; 27 ; 12 ) . We illustrate this with a simple example involving two clients with different data distributions and a linear predictor in Figure 1 . Similar problems can arise with min-max fairness ( 16 ) , which optimizes for the worst-off client but may still compromise overall global performance. Figure 1: In this example, we use the AFL algorithm to enforce loss parity between the two clients, and our approach, EAGLE , to enforce loss gap parity. Enforcing loss parity favors \text{client}~1 , which has noisier data and a more complex prediction task, resulting in poorer performance for \text{client}~0 . In contrast, optimizing for loss gap parity handles this imbalance more effectively. To address these issues, we adopt a fairness objective centered on encouraging loss gap parity across participating clients in federated learning. The loss gap of a client is defined as the difference between the loss of the global model and the lowest loss achievable using only the local data of that client. We consider a global model fair when these gaps are similar across clients, ensuring that all clients benefit relatively equally from collaboration. To achieve this, we introduce EAGLE, a novel algorithm that augments FedAvg with a regularization term explicitly penalizing the variance of client loss gaps. By tuning the regularization weight, EAGLE allows one to trade off overall accuracy against fairness, consistently reducing disparities in relative improvements compared to optimal local model of each client. Theoretically, we provide convergence guarantees for EAGLE under non-convex local losses, showing that it reaches a saddle point of the regularized objective. We further connect these guarantees to the standard federated learning objective via a novel heterogeneity measure, highlighting how greater client heterogeneity can reduce global utility when enforcing fairness. Empirically, we demonstrate the effectiveness of EAGLE on synthetic data as well as the EMNIST and DirtyMNIST benchmarks, with both linear and convolutional models. Our results show that EAGLE consistently reduces the variance in loss gaps across clients while maintaining comparable or improved overall performance relative to FedAvg and strong loss parity and min-max fairness baselines. Related work. Several approaches in the literature define fairness in federated learning through client losses, either by reducing disparities in losses across clients or by optimizing for the worst-case loss ( 10 ; 9 ; 16 ) . The motivation is that a fairer model should perform evenly across clients. However, in heterogeneous settings where clients have different optimal local losses and misaligned optimal models, defining fairness solely based on absolute losses can undesirably penalize clients with easier data, effectively forcing them to perform as poorly as clients with noisier or more complex data.

Read the complete text