Leveraging MoE-based Large Language Model for Zero-Shot Multi-Task Semantic Communication

Sin-Yu Huang, Renjie Liao, and Vincent W.S. Wong · 2025

Read Leveraging MoE-based Large Language Model for Zero-Shot Multi-Task Semantic Communication by Sin-Yu Huang, Renjie Liao, and Vincent W.S. Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15722.txt.

Introduction

Sin-Yu Huang, Renjie Liao, and Vincent Wong propose, in twenty twenty-five, a semantic communication system that can take on new tasks with no retraining. Semantic communication sends meaning, not raw bits, to save wireless resources. Old multi-task systems need task-specific embeddings and full retraining for each new task. Their design leans on FLAN, an instruction-tuned language model known for zero-shot skill. They add a mixture-of-experts layer, where different sub-networks handle different inputs, to the FLAN model.

A feature extraction module adapts to each task and to the signal-to-noise ratio. The result, MoE-FLAN-SC, improves on the base model at no extra compute cost, they report. Simulations show it beats three state-of-the-art models on average accuracy across four unseen tasks.

Catalog record

CREATORSin-Yu Huang, Renjie Liao, and Vincent W.S. Wong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15722

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging MoE-based Large Language Model for Zero-Shot Multi-Task Semantic Communication Sin-Yu Huang, Renjie Liao, and Vincent W.S. Wong Abstract Multi-task semantic communication (SC) can reduce the computational resources in wireless systems since retraining is not required when switching between tasks. However, existing approaches typically rely on task-specific embeddings to identify the intended task, necessitating retraining the entire model when given a new task. Consequently, this drives the need for a multi-task SC system that can handle new tasks without additional training, known as zero-shot learning. Inspired by the superior zero-shot capabilities of large language models (LLMs), we leverage pre-trained instruction-tuned LLMs, referred to as fine-tuned language net (FLAN), to improve the generalization capability. We incorporate a mixture-of-experts (MoE) architecture in the FLAN model and propose MoE-FLAN-SC architecture for multi-task SC systems. Our proposed MoE-FLAN-SC architecture can further improve the performance of FLAN-T5 model without increasing the computational cost. Moreover, we design a multi-task feature extraction module (FEM) which can adaptively extract relevant features across various tasks given the provided features and signal-to-noise ratio (SNR). Simulation results show that our proposed MoE-FLAN-SC architecture outperforms three state-of-the-art models in terms of the average accuracy on four different unseen tasks. Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, Canada Email: {syhuang, rjliao, vincentw}@ece.ubc.ca I Introduction Task-oriented semantic communication (SC) is an emerging technology for the sixth-generation (6G) wireless networks [ 1 ] . In task-oriented SC, the transmitter only sends the task-related features to the receiver. In [ 2 ] , Shao et al. proposed a learning-based task-oriented communication scheme that can efficiently extract features from images for image classification tasks. In [ 3 ] , Hu et al. studied the impact of semantic noise on task-oriented SC systems and introduced a feature importance module that can dynamically adjust the weights for different features. Nevertheless, the aforementioned works only study a single-task scenario. To perform a new task, the entire model needs to be retrained. In [ 4 ] , U-DeepSC is proposed to handle multiple tasks across three modalities: text, speech, and image. It can achieve performance comparable to that of single-task SC systems trained individually on each task. However, for each task, U-DeepSC relies on a task-specific embedding vector shared by the transmitter and receiver in order to identify the tasks to be performed. However, it limits the model’s generalization ability to tasks that have not been explicitly trained on, referred to as unseen tasks [ 5 ] . On the other hand, some recent works have proposed approaches to transfer knowledge from an existing task to an unseen task. In [ 6 ] , Leung et al. proposed a generalizable multi-task communication paradigm, where only the decoder needs to be retrained for unseen tasks. However, retraining the decoder, which is nearly half of the model size, is both time- and resource-intensive. Performing unseen tasks without either additional training or specific examples for those tasks is called zero-shot learning. In this context, large language models (LLMs) have shown to possess strong zero-shot ability. In [ 7 ] , Raffel et al. introduced a unified text-to-text transfer Transformer (T5) model, showcasing LLMs’ strong transfer learning capabilities across tasks such as summarization, question answering, and text classification. Building on the T5 model, Chung et al. in [ 8 ] proposed the fine-tuned language net (FLAN)-T5 model, demonstrating that instruction fine-tuning with a number of tasks can significantly enhance the model’s zero-shot performance. Despite the impressive results achieved by LLMs, their performance is at the expense of larger model sizes. In multi-task SC systems, another challenge arises due to gradient conflicts between tasks [ 9 ] . During the training process, the gradients of different tasks can point to different directions. Therefore, updating the model parameters based on gradients of one specific task can degrade the performance of other tasks. Recall that task-oriented SC transmits only the task-relevant features to complete the intended task. In [ 4 ] , U-DeepSC uses a separate feature selection module (FSM) to select task-oriented features for each task. This approach effectively avoids gradient conflicts since each module independently optimizes the feature selection process specific to its task. However, when the number of tasks increases, the number of FSMs increases linearly, resulting in a larger model size. Furthermore, when encountering an unseen task, a new FSM needs to be trained. For a multi-task SC system, we face the following questions: Q1: How to design a multi-task SC system that can effectively tackle those unseen tasks without retraining? Q2: How to satisfy the diverse task requirements and improve the performance in a multi-task SC system without increasing the computational cost? Fig. 1 : The system model of the proposed MoE-FLAN-SC architecture, which is trained on multiple natural language processing tasks, including sentiment analysis, question answering, and summarization, using natural language prompts. By training on a diverse range of tasks, the model enhances its ability to generalize to previously unseen tasks. To address the above questions, in this paper, we leverage the mixture-of-experts (MoE)-based LLM [ 10 ] in a multi-task SC system. The model aims to perform zero-shot learning for unseen tasks while filters out the task-irrelevant features for each task. The contributions of this paper are as follows: • We propose an MoE-FLAN-SC architecture that facilitates zero-shot learning for multi-task SC system. The proposed architecture integrates MoE in a pretrained FLAN-T5 model and uses a single multi-task feature extraction module (FEM) to adaptively filter out irrelevant features for all tasks, thereby reducing the transmission overhead. We propose a two-phase algorithm to instruction-tune the model using public pool of prompts (P3) [ 11 ] , which is a collection of datasets with prompt templates. • The proposed MoE-FLAN-SC architecture can perform zero-shot learning on various unseen tasks. By integrating MoE into our model, it improves zero-shot performance with the same floating point operations per second (FLOPs) per token. The multi-task FEM also uses the concept of MoE that includes multiple extractor modules to filter out irrelevant features under different signal-to-noise ratio (SNR). • We conduct simulations for various unseen tasks and compare our proposed MoE-FLAN-SC architecture with three state-of-the-art models, which are the FLAN-T5 [ 8 ] , LM-adapted T5 model [ 7 ] , and a conventional approach that uses 8-bit unicode transformation format (UTF-8) encoding and Turbo coding. Simulation results show that our proposed MoE-FLAN-SC architecture achieves a higher average accuracy than the baseline models. It achieves a 3.83% increase in average accuracy under high SNR conditions without increasing the FLOPs per token. We also conduct ablation study under different numbers of experts and extractor modules in FEM. This paper is organized as follows. Section II introduces the system model and presents the tasks and prompts considered in our model. Section III presents the architecture of the proposed model for each component, including the encoder, FEM, and decoder. We also present the two-phase training algorithm. Results for performance evaluation and comparison are presented in Section IV . Conclusion is given in Section V . Notations : We use upper-case boldface letters to denote matrices and lower-case boldface letters to denote column vectors. \mathbb{R} and \mathbb{C} denote the sets of real and complex numbers, respectively.

Read the complete text