Smoothing ADMM for Non-convex and Non-smooth Hierarchical Federated Learning
Reza Mirzaeifard, Stefan Werner · 2025
Read Smoothing ADMM for Non-convex and Non-smooth Hierarchical Federated Learning by Reza Mirzaeifard, Stefan Werner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08869.txt.
Introduction
Reza Mirzaeifard and Stefan Werner adapt a classic optimization method for federated learning in this 2025 paper. Federated learning trains models across many sites without pooling data, and in hierarchical setups the sites cluster under layers, with messy, non-smooth goals that defeat standard methods.
Their framework extends the ADMM method with smoothing and, unlike most rivals, allows sites to update asynchronously and several times per round. Each layer can carry its own penalty function matched to local knowledge. Total variation penalties pull layers toward consensus, while non-convex penalties let them personalize. Experiments showed faster convergence and better accuracy than conventional approaches across varied scenarios.
Catalog record
| CREATOR | Reza Mirzaeifard, Stefan Werner |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08869
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Smoothing ADMM for Non-convex and Non-smooth Hierarchical Federated Learning Reza Mirzaeifard, Stefan Werner Abstract This paper presents a hierarchical federated learning (FL) framework that extends the alternating direction method of multipliers (ADMM) with smoothing techniques, tailored for non-convex and non-smooth objectives. Unlike traditional hierarchical FL methods, our approach supports asynchronous updates and multiple updates per iteration, enhancing adaptability to heterogeneous data and system settings. Additionally, we introduce a flexible mechanism to leverage diverse regularization functions at each layer, allowing customization to the specific prior information within each cluster and accommodating (possibly) non-smooth penalty objectives. Depending on the learning goal, the framework supports both consensus and personalization: the total variation norm can be used to enforce consensus across layers, while non-convex penalties such as minimax concave penalty (MCP) or smoothly clipped absolute deviation (SCAD) enable personalized learning. Experimental results demonstrate the superior convergence rates and accuracy of our method compared to conventional approaches, underscoring its robustness and versatility for a wide range of FL scenarios. I Introduction The rapid expansion of cyber-physical systems within the internet of things (IoT) ecosystem has necessitated a shift from centralized data processing to federated frameworks. This transition is driven by the significant computational demands, bandwidth constraints, and privacy concerns associated with traditional centralized methods [ 1 , 2 ] . Federated learning (FL) has emerged as a pivotal solution, enabling collaborative model training across distributed devices while keeping data localized, thereby addressing critical privacy challenges and communication overhead [ 3 , 4 ] . Recent advancements have further enhanced FL’s scalability through hierarchical architectures, where two layers of aggregation are performed: edge servers handle intra-set computations for individual client groups, while a cloud server manages inter-set coordination across clusters [ 5 , 4 ] . This hierarchical approach optimizes resource utilization and facilitates parallel computations, making it indispensable for large-scale distributed systems. Despite these benefits, hierarchical FL environments face substantial heterogeneity among participating clients, necessitating flexible update strategies. First, data heterogeneity arises from distribution skew, label skew, feature skew, and variations in data quality (e.g., quantity skew and noise levels) [ 6 , 7 ] . Second, model heterogeneity occurs when clients must tailor model architectures or optimizers to accommodate various hardware or domain requirements [ 8 , 9 ] . Third, task heterogeneity emerges when clients solve distinct yet related problems [ 10 , 11 ] . Fourth, communication heterogeneity stems from inconsistent network conditions and bandwidth constraints [ 12 , 13 ] . Finally, device heterogeneity introduces disparities in computational power and storage capacities [ 14 , 15 ] . These factors underscore the need for a flexible regularization strategy, support of asynchronous updates (to handle communication delays), and multiple updates per iteration (to leverage resource-rich clients) in hierarchical FL. In addition to these challenges, non-smooth and non-convex formulations have gained increasing importance in FL due to real-world complexities such as noise, outliers, and the need for personalization [ 16 , 17 , 18 , 19 ] . Non-smooth objectives offer robustness against outliers and heavy-tailed distributions, improving the reliability of the learned models. Non-smooth regularization can induce sparsity (e.g., using an l_{1} penalty) or similarity constraints (e.g., total variation norms) to promote interpretable structures [ 17 , 20 ] . Meanwhile, non-convex regularizers like smoothly clipped absolute deviation (SCAD) [ 21 ] and minimax concave penalty (MCP) [ 22 ] mitigate the bias of convex penalties and enable personalized learning by capturing intricate data relationships across heterogeneous clients [ 16 ] . However, state-of-the-art methods often struggle with non-convexity and non-smoothness, especially when asynchronous updates, multiple local iterations, and scalability are required [ 23 , 24 ] . Consequently, there is a need for a new optimization framework that efficiently manages these complexities to produce robust, interpretable, and personalized solutions for large-scale hierarchical FL environments. This work introduces the hierarchical federated smoothing ADMM (HFSAD), a framework tailored for hierarchical FL that addresses non-convex, non-smooth optimization challenges while accommodating heterogeneous network and computational conditions. HFSAD leverages smoothing techniques selectively on consensus or personalization penalties (e.g., total variation norms), making them tractable within an ADMM-based scheme. At the same time, it retains non-smooth penalties (e.g., l_{1} , MCP, SCAD) for sparsity and inductive knowledge about parameters, thereby promoting interpretability and robust regularization. This dual approach balances global alignment and local customization, while enabling principled handling of outliers or heavy-tailed data distributions. HFSAD supports asynchronous updates and multiple local iterations, accommodating heterogeneous network and computational conditions. By uniting smoothing-driven linearization with ADMM’s decomposition capabilities, our method maintains scalability across distributed clusters, ensuring robustness to outliers, interpretability via sparse solutions, and convergence guarantees. Experimental results demonstrate that HFSAD surpasses existing approaches in accuracy, speed, and flexibility, establishing a new standard for distributed data analysis under diverse system constraints. Mathematical Notations : Scalars are denoted by lowercase letters, column vectors by bold lowercase letters, and matrices by bold uppercase letters. The transpose of a matrix \mathbf{A} is indicated by \mathbf{A}^{\top} . The j th column of \mathbf{A} is denoted by \mathbf{a}_{j} , and the element in the i th row and j th column of \mathbf{A} is denoted by a_{ij} . The proximal operator of a function f(\cdot) , scaled by a parameter \gamma , is defined as \textbf{Prox}_{f}\mathopen{}\left(w;\gamma\right)\mathclose{}=\argmin_{x}\mathopen{}\left\{f(x)+\frac{1}{2\gamma}\left\|x-w\right\|_{2}^{2}\right\}. Finally, \partial f(u) denotes the sub-gradient of the function f(\cdot) at point u . II Problem Formulation Assume that the system consists of one cloud server and L edge servers, where each edge server, indexed by l with l=1,2,\ldots,L , is associated with a unique set of clients V_{l} . Each set V_{l} contains N_{l} clients, and every client j in V_{l} possesses its own local dataset D_{lj}=\{\mathbf{X}_{lj},\mathbf{y}_{lj}\}. Consequently, the global objective function is defined as follows: \min_{\{\{\mathbf{w}^{j}_{l}\}_{j\in\mathcal{N}_{l}}\}_{l=1}^{L}}\sum_{l=1}^{L}\sum_{j\in V_{l}}f_{lj}(\mathbf{w}^{j}_{l})+\mathcal{R}_{c}(\{\mathbf{w}^{j}_{l}\})+\mathcal{R}_{w}(\{\mathbf{w}^{j}_{l}\}), (1) where \{\{\mathbf{w}^{j}_{l}\}_{j\in\mathcal{N}_{l}}\}_{l=1}^{L} denotes the local model parameters across clients. Further, within each cluster V_{l} , there is a cluster-level parameter \mathbf{w}_{l} to which individual client parameters are aligned. The regularization function \mathcal{R}_{c}(\cdot) is designed to enforce global consensus or allow for personalization within a cluster, while \mathcal{R}_{w}(\cdot) imposes prior knowledge about the parameters (e.g., based on historical data or domain expertise). We now extend our formulation by introducing a set of cluster-level parameters \{\mathbf{w}_{l}\}_{l=1}^{L} and an overall global parameter \mathbf{w}_{0} . With these additional parameters, the objective function is rewritten as: \min_{\mathbf{W}}\quad\sum_{l=1}^{L}\sum_{j\in V_{l}}f_{lj}(\mathbf{w}^{j}_{l})+\mathcal{R}_{c}\Bigl(\{\mathbf{w}^{j}_{l}\}_{j=1}^{N_{l}},\mathbf{w}_{l}\Bigr)\\[2.84526pt] +\,\mathcal{R}_{c}\Bigl(\{\mathbf{w}_{l}\}_{l=1}^{L},\mathbf{w}_{0}\Bigr)+\mathcal{R}_{w}(\mathbf{w}_{0})+\mathcal{R}_{w}\Bigl(\{\mathbf{w}_{l}\}_{l=1}^{L}\Bigr). (2) Here, \mathbf{W} denotes the collection of all parameters, including the local client parameters \{\mathbf{w}^{j}_{l}\} , the cluster-level parameters \{\mathbf{w}_{l}\}_{l=1}^{L} , and the overall global parameter \mathbf{w}_{0} . In this formulation, the first regularization term enforces intra-cluster relationships (which may include personalization), while the second term captures the relationship between the cluster-level parameters and the overall global parameter. To further refine the model, we decompose the regularization into per-cluster components by defining \mathcal{R}_{c}^{l}(\mathbf{w}_{l},\mathbf{w}_{0}), that quantifies the discrepancy between the cluster-level parameter \mathbf{w}_{l} and the global parameter \mathbf{w}_{0} for the l th cluster. Moreover, we allow the intra-cluster regularization to be flexible enough to enable personalization for each client by denoting it as \mathcal{R}_{c}^{l,j}(\mathbf{w}_{l}^{j},\mathbf{w}_{l}). This leads to the equivalent formulation: \min_{\mathbf{W}}\quad\sum_{l=1}^{L}\Biggl[\sum_{j\in V_{l}}f_{lj}(\mathbf{w}^{j}_{l})+\mathcal{R}_{c}^{l,j}(\mathbf{w}^{j}_{l},\mathbf{w}_{l})\\[2.84526pt] +\,\mathcal{R}_{c}^{l}(\mathbf{w}_{l},\mathbf{w}_{0})+\mathcal{R}_{w}^{l}(\mathbf{w}_{l})\Biggr]+\mathcal{R}_{w}^{0}(\mathbf{w}_{0}). (3) This formulation decouples the regularization terms, allowing separate control the intra-cluster consensus (or personalization) between local and cluster-level models (via \mathcal{R}_{c}^{l,j} ) and the inter-cluster relationship with the global model (via