An Information-Theoretic Analysis for Federated Learning under Concept Drift

Fu Peng, Meng Zhang, Ming Tang · 2025

Read An Information-Theoretic Analysis for Federated Learning under Concept Drift by Fu Peng, Meng Zhang, Ming Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21036.txt.

Introduction

Fu Peng, Meng Zhang, and Ming Tang analyze federated learning when data keep changing in a 2025 paper. Federated learning trains models across many devices without gathering raw data. Most studies assume fixed datasets, but real data arrive as streams whose patterns drift, and performance decays.

The authors model drift as a Markov chain and define a stationary generalization error for future data. They bound that error using standard information measures. They also propose an algorithm to reduce the loss from drift. The analysis matters for phone keyboards, sensors, and other systems that learn from ever-shifting streams.

Catalog record

CREATORFu Peng, Meng Zhang, Ming Tang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-21036

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Information-Theoretic Analysis for Federated Learning under Concept Drift Fu Peng, Meng Zhang, Ming Tang Abstract Recent studies in federated learning (FL) commonly train models on static datasets. However, real-world data often arrives as streams with shifting distributions, causing performance degradation known as concept drift. This paper analyzes FL performance under concept drift using information theory and proposes an algorithm to mitigate the performance degradation. We model concept drift as a Markov chain and introduce the Stationary Generalization Error to assess a model’s capability to capture characteristics of future unseen data. Its upper bound is derived using KL divergence and mutual information. We study three drift patterns (periodic, gradual, and random) and their impact on FL performance. Inspired by this, we propose an algorithm that regularizes the empirical risk minimization approach with KL divergence and mutual information, thereby enhancing long-term performance. We also explore the performance-cost tradeoff by identifying a Pareto front. To validate our approach, we build an FL testbed using Raspberry Pi4 devices. Experimental results corroborate with theoretical findings, confirming that drift patterns significantly affect performance. Our method consistently outperforms existing approaches for these three patterns, demonstrating its effectiveness in adapting concept drift in FL. I Introduction FL is a distributed machine learning approach that enables many clients to collaboratively train a neural network model [ 1 ] . Recent FL research primarily focuses on the design of incentive mechanisms [ 2 ] , improving FL efficiency [ 3 ] , utilization of computing sources [ 4 ] , etc. In these methods, clients perform training on static datasets to obtain the global model and achieve better utilization of computing and communication resources [ 1 ] . Fig. 1: An illustration of an FL system under concept drift. Fig. 2: Users’ interest in news topics changes over time. In practical FL systems, data is typically collected as data streams, whose statistical properties evolve over time. Fig. 1 illustrates an FL system under changing data distributions. For example, in real-world news recommendation scenarios such as the MIND dataset [ 5 ] , users’ interests in news topics shift over time (see Fig. 2 ). As user interests change, the characteristics of their behavior data also change, leading to a time-varying local data distribution. This drift requires models to be retrained or adapted to maintain performance on new and unseen data. This phenomenon is known as concept drift [ 6 ] . Generally, concept drift can typically be classified into three patterns: periodic, gradual, and random [ 6 , 7 ] . In periodic pattern, the changes of data distribution exhibit a cyclical nature, e.g., the time-varying traffic flow throughout the day. Gradual pattern characterizes slow shifts in data over time, such as changing trends in popular food or movies. Random pattern characterizes changes without any clear pattern, such as black swan events in the stock market. Concept drift has been widely studied in centralized machine learning. Early methods retrain models using recent data in a sliding window [ 8 , 9 ] , often discarding historical information. Others adapt decision trees via windowed updates or node pruning [ 10 , 11 ] , but these are limited to specific models. Neural network-based approaches [ 12 ] adjust model capacity by adding hidden nodes, which is computationally costly in FL. However, these centralized methods are not directly applicable to FL due to its decentralized architecture, limited communication bandwidth, and strict privacy constraints. Recent studies have addressed concept drift in FL with various strategies. Yoon et al. [ 13 ] and Yu et al. [ 14 ] focused on enhancing local model adaptability, while neglecting global model robustness. Jothimurugesan et al. [ 15 ] formulated drift as a clustering problem but incurred high client-side computation. Canonaco et al. [ 16 ] and Panchal et al. [ 17 ] proposed adaptive optimizers for faster adaptation, yet overlooked performance degradation. However, none of these works offer a theoretical foundation for analyzing drift in FL. To bridge this gap, we adopt an information-theoretic framework to analyze concept drift in FL. Compared to classical tools such as VC dimension [ 18 ] and PAC-Bayesian bounds [ 19 ] , information-theoretic analysis offers a more flexible and insightful way to characterize the generalization error under dynamic and distributed settings. Bu etal. [ 20 ] derived an information-theoretic upper bound on the generalization error of supervised learning algorithms, while Wu etal. [ 21 ] extended such analysis to transfer learning. In this work, we build upon these results and extend them to FL under concept drift. Based on this theoretical foundation, it is crucial to quantitatively evaluate the performance of an FL system under concept drift, with a focus on deriving performance bounds using information theory. The performance bound reveals how factors such as drift patterns and the magnitude of distributional changes influence the generalization capability of the system. Moreover, it provides insights into how to design algorithms that mitigate performance degradation. In practice, however, improving performance often comes with increased communication costs, computational overhead, or training efforts. Therefore, it is also important to understand the tradeoffs between system performance and cost. In this work, our objective is to address the following key questions. Question 1 (Performance Metric and Its Bound) . How should we design the performance metric and establish its bound to evaluate an FL system under concept drift? Designing an appropriate performance metric is challenging because the performance metric should be able to simultaneously evaluate the inference accuracy of the FL global model on the current data distribution and reflect its generalization ability 1 1 1 The generalization ability of a machine learning model refers to its capability to perform well and make accurate predictions on new and previously unseen data points that were not part of the training dataset [ 20 ] . to future unseen data distributions, which are highly dynamic and unpredictable. Furthermore, the performance metric should also capture the changes in data distribution for the long term. To address these challenges, we introduce a performance metric termed the Stationary Generalization Error by employing a Markov chain to model the evolution of data distributions over time. To establish the bound for the performance metric, we provide an information-theoretic bound expressed in terms of mutual information and KL divergence. This is challenging because the generalization error of the global model must be derived from local models in the FL aggregation process. This difficulty arises due to the decentralized nature of FL, where local models are trained independently and share only partial information. Consequently, quantifying how the aggregation process influences the global model’s generalization performance becomes non-trivial. Question 2 (Algorithm Design) . How can we formulate an algorithm to alleviate the performance decline resulting from concept drift? To address Question 2 , we propose an algorithm inspired by the information-theoretic performance bounds derived in our study. This algorithm enhances the ERM approach [ 22 ] by incorporating regularization terms based on mutual information and KL divergence. The objective is to mitigate performance degradation caused by concept drift and to enhance the system’s long-term average performance. Question 3 (Performance-cost Tradeoff) . What is the tradeoff (and hence the Pareto front) between FL system performance

Read the complete text