Multi-task Federated Learning with Encoder-Decoder Structure: Enabling Collaborative Learning Across Different Tasks

Jingxuan Zhou, Weidong Bao, Ji Wang, Dayu Zhang, Xiongtao Zhang, Yaohong Zhang · 2025

Read Multi-task Federated Learning with Encoder-Decoder Structure: Enabling Collaborative Learning Across Different Tasks by Jingxuan Zhou, Weidong Bao, Ji Wang, Dayu Zhang, Xiongtao Zhang, Yaohong Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09800.txt.

Introduction

Federated learning lets scattered devices train one model without pooling their data, but it usually demands that every client run the same kind of model on the same kind of task. Jingxuan Zhou and co-authors loosen that rule with M-Fed, a framework built on the encoder-decoder shape that many modern models share. Clients keep their own decoders for their own tasks while the shared encoder carries general knowledge between them. A loss term pulls each local decoder toward the global one as training proceeds.

The design stays light and modular, bolting onto standard federated training rather than replacing it. On two benchmark datasets, M-Fed let unlike clients learn together and beat the traditional single-task approach. The authors point toward federated systems where clients with different jobs can still teach each other.

Catalog record

CREATORJingxuan Zhou, Weidong Bao, Ji Wang, Dayu Zhang, Xiongtao Zhang, Yaohong Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-09800

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-task Federated Learning with Encoder-Decoder Structure: Enabling Collaborative Learning Across Different Tasks Jingxuan Zhou, Weidong Bao, Ji Wang, Dayu Zhang, Xiongtao Zhang, Yaohong Zhang Abstract Federated learning has been extensively studied and applied due to its ability to ensure data security in distributed environments while building better models. However, clients participating in federated learning still face limitations, as clients with different structures or tasks cannot participate in learning together. In view of this, constructing a federated learning framework that allows collaboration between clients with different model structures and performing different tasks, enabling them to share valuable knowledge to enhance model efficiency, holds significant practical implications for the widespread application of federated learning. To achieve this goal, we propose a multi-task federated learning with encoder-decoder structure (M-Fed). Specifically, given the widespread adoption of the encoder-decoder architecture in current models, we leverage this structure to share intra-task knowledge through traditional federated learning methods and extract general knowledge from the encoder to achieve cross-task knowledge sharing. The training process is similar to traditional federated learning, and we incorporate local decoder and global decoder information into the loss function. The local decoder iteratively updates and gradually approaches the global decoder until sufficient cross-task knowledge sharing is achieved. Our method is lightweight and modular, demonstrating innovation compared to previous research. It enables clients performing different tasks to share general knowledge while maintaining the efficiency of traditional federated learning systems. We conducted experiments on two widely used benchmark datasets to verify the feasibility of M-Fed and compared it with traditional methods. The experimental results demonstrate the effectiveness of M-Fed in multi-task federated learning. keywords Multi-task learning, Federated Learning, Encoder-Decoder structure, Model Heterogeneity 1 Introduction In recent decades, deep neural networks have been widely applied in numerous real-world scenarios due to their remarkable success in solving complex problems across different domains. For instance, as stated in Zhou (2016) , tasks such as recognition, segmentation, and image parsing constitute the core components of medical image analysis. In the field of autonomous driving, to navigate safely in the surrounding environment, it is imperative to detect and analyze all objects in the scene, accurately locate these objects, and estimate their distances and trajectories Vandenhende (2022) . The continuous surge of other innovative applications is driving an exponential growth in the number of machine learning models specifically designed to meet diverse demands. Nowadays, it has become the norm to construct specific deep learning models for various types of tasks. Currently, although large language models capable of executing diverse tasks have emerged, due to the limited computational capabilities of most devices, a more prevalent practice in practical applications is still to customize exclusive models for specific types of tasks. However, building deep learning models for specific tasks requires extensive training datasets and computational resources, posing considerable challenges for individual users. To enable individual users to build efficient models even with limited training data and computational resources, Federated Learning (FL) is proposed as a solution, which can also effectively protect the privacy of user data Konečný et al. (2015) . Despite its numerous advantages, FL also exhibits certain limitations in practical applications, primarily manifested in two aspects: data scarcity and model constraints: 1. data scarcity : For tasks with high specificity and a limited number of available clients, FL faces difficulties in acquiring sufficient high-quality data from a restricted number of clients during implementation. This leads to constraints on improving model performance. Therefore, traditional FL methods still encounter challenges in constructing high-performance models for specific tasks with scarce data. 2. model constraints : Traditional FL methods require all clients to perform the same task and possess the same model structure. This implies that when clients participate in the FL system, they must collaborate with a group of clients performing the same task and sharing the same model structure. Although knowledge distillation Hinton et al. (2015) techniques provide clients with the flexibility to operate using models with different structures, the application of FL in diverse tasks still faces significant challenges. The requirement for task homogeneity in FL limits its capacity to accommodate a larger number of clients. Our objective is to design a framework that addresses the limitations of traditional FL, enabling clients with diverse tasks and model structures to participate in FL collectively, thereby naturally mitigating the issue of data scarcity. We observe that the current model construction process tends to be streamlined, commonly adopting an encoder-decoder architecture. In this architecture, the encoder is responsible for converting variable-length input sequences into fixed-length vectors, while the decoder performs the opposite operation, generating variable-length sequences from fixed-length vectors. The encoder, as a fundamental component for feature extraction, demonstrates adaptability to various downstream tasks; by contrast, the decoder is customized for specific downstream tasks to better meet their professional requirements. By combining a unified encoder with different decoders, multitask adaptation can be achieved under similar network architectures. We propose a method named M ulti-task F ederated Learning with E ncoder- D ecoder Structure (M-Fed). This method aims to break through the inherent limitations of data scarcity and model constraints in traditional FL, enabling clients with different tasks and diverse model structures to collaborate and learn from each other, thereby effectively alleviating the issue of scarce task data. As a lightweight and scalable technology, M-Fed allows clients to participate in FL without being restricted by model structures and task types, simply by making local adjustments to their original models. Furthermore, to enhance training efficiency, we introduce a variable threshold mechanism into the actual training process. This mechanism can effectively mitigate the negative impact of the global model on local models during the initial stage of training. The differences between previous studies and this research are shown in Figure 1 . Previous studies primarily focused on collaborative learning among clients with identical models, while the M-Fed proposed in this research faces two unique challenges in collaborative learning: 1) constructing a framework to enable collaborative learning among multi-task clients; 2) effectively transferring knowledge between heterogeneous models. To address these challenges, we have designed a multi-task collaborative learning framework aimed at minimizing the differences between heterogeneous models and achieving knowledge transfer without compromising the execution of local tasks. The main contributions and innovations of this paper can be summarized as follows: Figure 1: Comparison of multi-task FL with other typical methods. Different colors represent different types of tasks. a. represents the case where local data is used for training models and there is no interaction between clients, with the quality of the models entirely reliant on the quality of the local data. b. demonstrates the restriction of traditional FL that prohibits cross-learning between client models with diverse structures executing various tasks. c. indicates the M-Fed method can transfer general knowledge to improve performance across different tasks, even if they have different model architectures on multiple clients. • We propose the M-Fed method, a FL technique tailored for multi-task scenarios. This method enables cooperative learning across heterogeneous clients for different tasks, while retaining the privacy-preserving advantages of traditional FL approaches. Clients only need to make local adjustments to their original models to participate in M-Fed, without altering their original tasks. • The M-Fed method is based on the widely used encoder-decoder

Read the complete text