GP-FL: Model-Based Hessian Estimation for Second-Order Over-the-Air Federated Learning

Shayan Mohajer Hamidi, Ali Bereyhi, Saba Asaad · 2024

Read GP-FL: Model-Based Hessian Estimation for Second-Order Over-the-Air Federated Learning by Shayan Mohajer Hamidi, Ali Bereyhi, Saba Asaad (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03867.txt.

Introduction

Shayan Mohajer Hamidi, Ali Bereyhi, and co-authors give a way to train shared models across wireless links with second order gains, in a 2024 paper. Shared training lets phones learn as a group with no raw data shared. Fast methods need curve maps from each phone, but sending them costs far too much.

Their trick reads the curve map at the server from noisy slopes sent through air. A flexible curve model and the known channel shape guide that read. Tests on many data sets beat both plain and fast past shared methods. The rate proof shows a clean blend of fast and sure steps.

Catalog record

CREATORShayan Mohajer Hamidi, Ali Bereyhi, Saba Asaad
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03867

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GP-FL: Model-Based Hessian Estimation for Second-Order Over-the-Air Federated Learning Thanks: Shayan Mohajer Hamidi is with the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON N2L 3G1, Canada (email: [email protected]). Ali Bereyhi is with the Department of Electrical and Computer Engineering (ECE), University of Toronto, Toronto, M5S 2E4, Canada (e-mail: [email protected]). Saba Asaad is with the Department of Electrical Engineering and Computer Science, York University, Toronto, ON M3J 1P3, Canada (e-mail: [email protected]). H. Vincent Poor is with the Department of Electrical and Computer Engineering (ECE), Princeton University, Princeton, NJ 08544 USA (e-mail: [email protected]). Shayan Mohajer Hamidi, Ali Bereyhi, Saba Asaad Abstract Second-order methods are widely adopted to improve the convergence rate of learning algorithms. In federated learning (FL), these methods require the clients to share their local Hessian matrices with the parameter server (PS), which comes at a prohibitive communication cost. A classical solution to this issue is to approximate the global Hessian matrix from the first-order information. Unlike in idealized networks, this solution does not perform effectively in over-the-air FL settings, where the PS receives noisy versions of the local gradients. This paper introduces a novel second-order FL framework tailored for wireless channels. The pivotal innovation lies in the PS’s capability to directly estimate the global Hessian matrix from the received noisy local gradients via a non-parametric method: the PS models the unknown Hessian matrix as a Gaussian process, and then uses the temporal relation between the gradients and Hessian along with the channel model to find a stochastic estimator for the global Hessian matrix. We refer to this method as G aussian p rocess-based Hessian modeling for wireless FL (GP-FL) and show that it exhibits a linear-quadratic convergence rate. Numerical experiments on various datasets demonstrate that GP-FL outperforms all classical baseline first and second order FL approaches. I Introduction Traditionally, the training of machine learning (ML) models has followed a centralized approach, with the training data residing in a data center or cloud parameter server (PS). However, in numerous contemporary applications, there is a growing reluctance among devices to share their private data with a remote PS. Addressing this challenge, federated learning (FL) has emerged as a viable solution [ 1 ] . FL allows devices to actively contribute to the training of a global model by leveraging only their local datasets while being facilitated by a central PS. In the FL framework, devices transmit solely their local updates to the PS, thus sidestepping the need to disclose raw datasets. The process unfolds in several steps: initially, the PS disseminates the current global model parameters to all participating devices. Subsequently, each device conducts local model training based on its unique dataset and transmits the resulting local updates back to the PS. In the final step, the PS aggregates these local updates, producing a new global model parameters for the subsequent iteration of distributed training. Considering the large sizes of the model updates, the iterative communication between the PS and clients incurs a significant communication cost. This can significantly decelerate the convergence of FL, since the communication are typically rate-limited in practice [ 2 , 3 , 4 ] . Several lines of research have been dedicated to theoretically characterizing the fundamental trade-offs between learning performance and communication cost, as well as improving the communication efficiency, within the FL framework [ 5 , 6 ] . I-A Communication-Efficient FL Early efforts to improve the communication efficiency in the FL framework can be categorized roughly into two approaches: ( i ) the first approach tries to reduce the communication overhead per round. One way to achieve this involves utilizing quantization [ 7 ] and sparsification [ 8 ] techniques. These techniques aim to diminish the transmitted bits and eliminate redundant parameter updates. In general, these approaches come with the trade-off of potentially degrading the model performance, and they must take into account the compatibility for the aggregation operation in FL [ 9 ] . ( ii ) The second approach intends to minimize the total communication rounds. The most well-known example is the federated averaging (FedAvg) algorithm, in which clients execute multiple local iterations before sharing their models with the PS [ 1 ] , which can significantly reduce the total number of rounds. Some variants of the gradient descent (GD) method have further demonstrated the ability to substantially reduce the overall communication rounds compared to the naive GD [ 10 , 11 ] . These previous approaches ignore the properties of the communication network, and look at the communication links as rate-limited orthogonal noiseless channels. Nevertheless, later studies have shown that using the properties of the underlying network, further gain in terms of communication efficiency can be achieved [ 2 ] . A well-known case is wireless FL, in which clients can use the superposition property of multiple access channel (MAC) to perform the computation directly over the air and significantly reduce the communication costs [ 12 ] . In fact, in wireless networks the abstract view on communication links corresponds to the conventional digital transmission, where coded symbols are sent using techniques such as orthogonal frequency-division multiplexing (OFDM). For wireless FL, however, the clients can send their local parameters simultaneously via analog transmission, and the PS can apply the concept of over-the-air computation (AirComp) to aggregate the global parameter. This method allows aggregation to be performed directly over the air, significantly reducing the required bandwidth [ 12 , 13 ] . This approach has gained traction in wireless FL as an effective strategy for mitigating communication costs [ 14 , 15 , 16 , 17 , 18 , 19 ] . I-B Second-Order Wireless FL A standard framework for wireless FL consider first-order algorithms, where the clients transmit their local gradients, and the PS aggregates them gradient directly over the air using the AirComp technique. We refer to this framework as first-order AirComp . Similar to other first-order methods, first-order AirComp at best achieves linear convergence, which leads to a relatively large number of iteration rounds to reach the desired accuracy. We further note that the analog nature of AirComp-aided computation leads to extra aggregation error from the channel. This can be seen as zero-mean fluctuations, which adds to the noiseless aggregated gradient, i.e., the estimator that PS computes in perfect FL settings. As a result, the estimator of global gradient computed in first-order AirComp has higher variance as compared with FL with noiseless aggregation. This leads to further degradation in terms of convergence behavior. One potential solution is to incorporate second-order methods, e.g., Newton-type methods, into the wireless FL setting, as explored in [ 20 ] . Unlike first-order methods, second-order approaches benefit from a quadratic convergence rate by computing the update direction using both first and second-order derivatives of the loss function. The update direction computed in these methods is often called Newton direction . Integrating second-order information into FL requires further access to the second-order information at the PS. This information is obtained by clients sharing their local Hessian matrices with the PS, which imposes a significant communication burden, especially in wireless settings with strictly-restricted links. To mitigate this issue, various studies have explored ways to approximate the Hessian matrix from the first-order

Read the complete text