Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization
Li Shen [email protected], Yan Sun [email protected], Dacheng Tao [email protected] · 2026
Read Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization by Li Shen [email protected], Yan Sun [email protected], Dacheng Tao [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12768.txt.
Introduction
Li Shen and two colleagues tackle client drift in federated learning with a new method, FedInit. Clients train locally on varied data, so their optima disagree and the shared model drifts. FedInit starts each local stage from a relaxed state moved away from the global state. Theory shows optimization error barely reacts to inconsistency, while generalization bears the brunt. Experiments match strong benchmarks with no extra training or communication cost. The relaxed start also plugs into other advanced methods for higher generalization.
Catalog record
| CREATOR | Li Shen [email protected], Yan Sun [email protected], Dacheng Tao [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105315 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.12768
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization Li Shen [email protected], Yan Sun [email protected], Dacheng Tao [email protected] Abstract Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the heterogeneous dataset. Previous works have implicitly studied that FL suffers from the “client-drift” problem, which is caused by the inconsistent optimum across local clients. However, till now it still lacks solid theoretical analysis to explain the impact of this local inconsistency. To alleviate the negative impact of “client drift” and explore its substance in FL, in this paper, we first propose an efficient FL algorithm FedInit , which allows employing the personalized relaxed initialization state at the beginning of each local training stage. Specifically, FedInit initializes the local state by moving away from the current global state towards the reverse direction of the latest local state. Moreover, to further understand how inconsistency disrupts performance in FL, we introduce the excess risk analysis and study the divergence term to investigate the test error in FL. Our studies show that optimization error is not sensitive to this local inconsistency, while it mainly affects the generalization error bound. Extensive experiments are conducted to validate its efficiency. The proposed FedInit method could achieve comparable results compared to several advanced benchmarks without any additional training or communication costs. Meanwhile, the stage-wise personalized relaxed initialization could also be incorporated into several current advanced algorithms to achieve higher generalization performance in the FL paradigm. † † shortheadings: Rethinking the Personalized Relaxed Initialization / Rethinking the Personalized Relaxed Initialization † † firstpage: 1 keywords Federated learning, personalized relaxed initialization, excess risk. 1 Introduction Since 30 developed federated learning (FL), it has become a promising paradigm to effectively make full use of edged computational powers of local devices. 19 further classify the different modes in FL based on the specific tasks and unique environmental setups. Different from centralized training, FL utilizes a central server to coordinate the clients to perform several local training stages and aggregate local models as one global model regularly. This creative training framework greatly improves the effective use of edged devices. However, it also has unavoidable disadvantages. Due to the local heterogeneous dataset, it usually suffers from the significant performance degradation. Several previous studies explore the essence of performance limitations and inherent biases in FL, and summarize them as the “client-drift” problems ( 1 ; 21 ; 25 ; 43 ; 52 ; 55 ; 57 ) . From the global objective perspective, 21 claim that the aggregated local optimum is always far away from the global optimum due to the local heterogeneity, which introduces the issue of w^{\star}\neq\frac{1}{C}\sum_{i}w_{i}^{\star} in FL. However, when the local training iterations are limited, local clients could not exactly achieve their local optimum. To describe this negative impact more accurately, 1 and 52 point out that each locally optimized objective should be regularized to be aligned with the global objective. Moreover, beyond the requirement of the local consistent objectives, 55 also indicates that the performance degradation could be mitigated when each local updates maintain high consistency at each communication round, which is more similar to the centralized scenarios. These explorations intuitively provide the forward-looking insights of improving the performance in FL. Although this analysis explains the inherent biases across local heterogeneous datasets, there is still no solid theoretical support to further understand the incomprehensible impact of the consistency, which also greatly hinders its developments. Therefore, our work tries to provide a new perspective on consistency. Motivation. From the training strategies, the largest difference between FL and general centralized training is to adopt local training processes, which results in local clients tilt towards the local optimum. We spontaneously and naturally expect to add an inverse effect in the local training process to correct the deviation. Several previous studies have introduced more variables, which has caused great pressure on communication bottlenecks. To further alleviate the negative impact of the “client-drift” problem and strengthen consistency in the FL paradigm without any additional communication burdens, in this paper, we take into account adopting the personalized relaxed initialization at the beginning of each communication round, dubbed FedInit method. Specifically, FedInit initializes each local models of active clients by moving away from the current global state towards the reverse direction of the current latest local state, which is named relaxed initialization (RI). Personalized RI helps each local model to revise its divergence and gather together with each other during the local training process. Because each initialization of the local model is more distant from the local optimum than the global model, locally trained solutions naturally maintain higher consistency under the same local training interval. This flexible approach is surprisingly effective in FL and only adopts one additional coefficient to control the divergence level of the initialization. It could also be easily incorporated as a plug-in into other advanced benchmarks to further improve their efficiency. We also provide a schematic illustration of the effects of this initialization. Moreover, to explicitly understand how and why RI could help to improve the performance in FL, we introduce the excess risk analysis to investigate the test error of FedInit under the smooth non-convex objectives, including the joint analysis on both the optimization error and the generalization error. Our theoretical analysis indicates that both optimization error and generalization error could be greatly improved by the RI technique. Furthermore, under PŁ -condition, RI could efficiently reduce the excess risk and the test error. Extensive empirical studies are conducted to validate the efficiency of the proposed FedInit method on different classical federated experimental setups. On the CIFAR-10 / 100 dataset, it could achieve SOTA results compared to several advanced benchmarks without additional costs. It also helps to enhance the consistency level in FL. Main contributions of our work are summarized as follows: • We propose an efficient and novel FL method, dubbed FedInit , which adopts the personalized relaxed initialization (RI) state on the selected local clients at each communication round. RI is dedicated to enhancing local consistency during training, and it is also a practical plug-in that could easily to incorporated into other methods. • One important contribution is that we introduce the excess risk analysis in the proposed FedInit method to understand the intrinsic impact of local consistency. Our theoretical studies prove that RI could help to improve both the optimization error and generalization error on the smooth and non-convex objectives. • Extensive numerical studies are conducted on the real-world dataset to validate the efficiency of the FedInit method, which could achieve the comparable results of several SOTA benchmarks, without any additional communication costs during the training. 2 Related Work Improving Consistency in FL. FL employs an enormous number of edge devices to jointly train a single model among the isolated heterogeneous dataset ( 19 ; 30 ) . As a standard benchmark, FedAvg ( 2 ; 30 ; 57 ) allows the local stochastic gradient descent (local SGD) ( 12 ; 26 ; 54 ) based updates and uniformly selected partial clients’ participation to alleviate the communication bottleneck. The stage-wise local training processes lead to significant divergence for each client ( 6 ;