Automatic Structured Pruning for Efficient Architecture in Federated Learning

Thai Vu Nguyen, Long Bao Le, Anderson Avila · 2024

Read Automatic Structured Pruning for Efficient Architecture in Federated Learning by Thai Vu Nguyen, Long Bao Le, Anderson Avila (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01759.txt.

Description

Abstract In Federated Learning (FL), training is conducted on client devices, typically with limited computational resources and storage capacity. To address these constraints, we propose an automatic pruning scheme tailored for FL systems. Our solution improves computation efficiency on client devices, while minimizing communication costs.

Catalog record

CREATORThai Vu Nguyen, Long Bao Le, Anderson Avila
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-01759

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automatic Structured Pruning for Efficient Architecture in Federated Learning Thai Vu Nguyen, Long Bao Le, Anderson Avila Abstract In Federated Learning (FL), training is conducted on client devices, typically with limited computational resources and storage capacity. To address these constraints, we propose an automatic pruning scheme tailored for FL systems. Our solution improves computation efficiency on client devices, while minimizing communication costs. One of the challenges of tuning pruning hyper-parameters in FL systems is the restricted access to local data. Thus, we introduce an automatic pruning paradigm that dynamically determines pruning boundaries. Additionally, we utilized a structured pruning algorithm optimized for mobile devices that lack hardware support for sparse computations. Experimental results demonstrate the effectiveness of our approach, achieving accuracy comparable to existing methods. Our method notably reduces the number of parameters by 89% and FLOPS by 90%, with minimal impact on the accuracy of the FEMNIST and CelebFaces datasets. Furthermore, our pruning method decreases communication overhead by up to 5x and halves inference time when deployed on Android devices. 1 1 1 The code implementation is available online: https://github.com/NguyenThaiVu/prune˙fl˙project . 1 INRS-EMT, University of Quebec 1 Introduction Recently, we have witnessed an increasing interest in shifting cloud computing to edge computing ( Li et al. 2021c ) . In fact, bringing computation to the edge of computer networks can reduce latency, benefiting real-time applications, such as autonomous driving ( Jin et al. 2024 ) . On the other hand, the data generated, at an unprecedented rate, from billions of edge devices can be used for training and improving AI models ( Jia et al. 2024 ; Nguyen and Le 2023 ) . In such scenarios, Federated Learning (FL) has emerged as a promising alternative to process such an extensive amount of data while preserving users’ privacy ( McMahan et al. 2017 ) . This is achieved by decentralizing the training process of machine learning (ML) models and keeping personal user data on clients’ devices. Thus, training is performed locally and only the model parameters are sent to the central server. The final global model is attained via aggregation of the parameters from all clients. In the FL settings, where the primary training process occurs on devices with limited computational and storage resources, there is growing interest in developing strategies to decrease the model’s footprint. Several studies, for instance, have proposed neural network pruning to reduce model size. The frameworks proposed in ( Li et al. 2021a ) and ( Isik et al. 2023 ) apply the lottery ticket hypothesis to the FL settings. The work presented in ( Qiu et al. 2022 ) selects the top-k weights of local models to be sent back to the central server for aggregation. In ( Bibikar et al. 2022 ) , authors propose a sparse training approach tailored specifically for the FL settings. A crucial limitation of such approaches is that they require pruning to be performed on the local devices. This results in higher computational costs when compared to the normal local training. Additionally, the differences in local models lead to different pruning masks, complicating the aggregation of these masks into the unified global model. To overcome these problems, we propose a pruning procedure that is executed on the central server rather than on individual local devices. It is important to note that the aforementioned works rely on unstructured pruning, which removes individual weight elements and results in a sparse network with many zero values. Performing fast computations on sparse matrices requires support from specialized libraries, such as cuSPARSE, or hardware (e.g., NVIDIA Ampere GPU) and clients’ devices participating in FL training often lack such requirements. Additionally, maintaining sparse data structures requires extra storage for information like compressed sparse rows or binary masks. Thus, inspired by the structured pruning technique presented in ( Li et al. 2017 ) , which prunes entire filters rather than individual weights, this paper proposes pruning filters in convolutional networks, maintaining the dense structure of the global model while ensuring compatibility with the simple computational capabilities of client devices. Figure 1: An overview of our pruning scheme in Federated Learning system. While centralized learning allows access to the training data and therefore proper tuning of pruning hyper-parameters, such as the sparsity fraction and the number of pruned layers, decentralized learning presents the inability to examine the whole training data directly during training. This makes it difficult to choose a suitable model architecture. For example, in the FL settings, we cannot perform a grid search to determine the optimal number of filters in a convolutional neural network (CNN). To address this challenge, we propose an automatic pruning algorithm, which dynamically defines the pruning boundaries and removes redundant filters without disrupting the FL procedure. The proposed framework for pruning CNNs in the FL settings is illustrated in Figure 1 . The FL training procedure is divided into two sub-stages. First, we perform automatic pruning to identify the well-suited architecture, as depicted in Figure 1 -a. This phase involves dynamically eliminating unnecessary filters to reduce the model’s size without yet considering the accuracy of the global model. After obtaining the final global model architecture, we proceed with the standard FL training phase (Figure 1 -b). The second stage is dedicated to training the final global model to achieve the desired accuracy for the specific task. Below, we summarize the four main contributions of our work: • We designed a pruning scheme for FL that significantly reduces redundant parameters/FLOPS in the global model while maintaining the desired accuracy. Additionally, the pruning procedure runs on the central server, reducing computational burdens for local clients. • We proposed a structured pruning algorithm that automatically determines the pruning boundary and preserves a dense network structure, ensuring fast computation on basic client hardware. • Extensive experiments show that our pruning algorithm is effective across various architectures, including CNNs, ResNet, and Inception. Additionally, the pruned models significantly reduce inference time on real-world Android devices. • We also present supplemental studies showing improvements in communication costs and consistency when training with varying random numbers of selected clients in the FL system. The rest of the paper is organized as follows. Section 2 presents the related work in network pruning, besides summarizing the differences between our paper and existing work. Section 3 introduces our proposed method to prune convolutional neural networks in the FL settings. In Section 4 , we conduct extensive experiments to validate the superior performance of our method, followed by the concluding remarks in Section 5 . 2 Related Works 2.1 Network Pruning in Federated Learning Pruning techniques are widely used to reduce the size of neural network models. These approaches are crucial to facilitate the execution of such models on resource-constrained devices (e.g., mobile and IoT gadgets). Pruning methods can be categorized into two main types: unstructured pruning ( Han et al. 2015 ) and structured pruning ( Li et al. 2017 ) . Unstructured pruning involves removing individual weights in the network, resulting in a sparse weight matrix. In contrast, structured pruning implies the elimination of entire components of the network, such as layers or filters, while maintaining a dense network architecture. Due to the decentralized nature of FL, applying unstructured pruning

Read the complete text