Mamba4Net: Distilled Hybrid Mamba Large Language Models For Networking PubID: pubid: 979-8-3315-0376-5/25/$31.00 © \copyright 2025 IEEE

Linhan Xia, Mingzhan Yang, Jingjing Wang, Ziwei Yan, Yakun Ren, Guo Yu, Kai Lei* · 2025

Read Mamba4Net: Distilled Hybrid Mamba Large Language Models For Networking PubID: pubid: 979-8-3315-0376-5/25/$31.00 © \copyright 2025 IEEE by Linhan Xia, Mingzhan Yang, Jingjing Wang, Ziwei Yan, Yakun Ren, Guo Yu, Kai Lei* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17147.txt.

Introduction

Linhan Xia and six co-authors shrink large language models for network tasks in this 2025 paper. Big transformer models are strong but slow and huge for tight gear. The team shifts network know-how into a leaner model shape. That shape runs in line time, not square time, as inputs grow.

Their frame trains the lean model to copy a big model's network skill. Tests cover view guess, stream rate choice, and job timing in clusters. The lean model beat non-model methods on all three tasks. It ran near four times the speed of big-model use. It also took a small share of the storage those methods need.

Catalog record

CREATORLinhan Xia, Mingzhan Yang, Jingjing Wang, Ziwei Yan, Yakun Ren, Guo Yu, Kai Lei*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17147

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mamba4Net: Distilled Hybrid Mamba Large Language Models For Networking PubID: pubid: 979-8-3315-0376-5/25/$31.00 © \copyright 2025 IEEE Linhan Xia, Mingzhan Yang, Jingjing Wang, Ziwei Yan, Yakun Ren, Guo Yu, Kai Lei* Abstract Transformer-based large language models (LLMs) are increasingly being adopted in networking research to address domain-specific challenges. However, their quadratic time complexity and substantial model sizes often result in significant computational overhead and memory constraints, particularly in resource-constrained environments. Drawing inspiration from the efficiency and performance of the Deepseek-R1 model within the knowledge distillation paradigm, this paper introduces Mamba4Net, a novel cross-architecture distillation framework. Mamba4Net transfers networking-specific knowledge from transformer-based LLMs to student models built on the Mamba architecture, which features linear time complexity. This design substantially enhances computational efficiency compared to the quadratic complexity of transformer-based models, while the reduced model size further minimizes computational demands, improving overall performance and resource utilization. To evaluate its effectiveness, Mamba4Net was tested across three diverse networking tasks: viewport prediction, adaptive bitrate streaming, and cluster job scheduling. Compared to existing methods that do not leverage LLMs, Mamba4Net demonstrates superior task performance. Furthermore, relative to direct applications of transformer-based LLMs, it achieves significant efficiency gains, including a throughput 3.96 times higher and a storage footprint of only 5.48% of that required by previous LLM-based approaches. These results highlight Mamba4Net’s potential to enable the cost-effective application of LLM-derived knowledge in networking contexts. The source code is openly available to support further research and development. I Introduction As global data traffic continues to surge exponentially, optimizing and controlling networking systems have become imperative to ensure seamless connectivity and performance. Deep learning technologies are at the forefront of this optimization, being employed to address a variety of networking tasks such as bandwidth prediction [ 1 ] , congestion control [ 2 ] [ 3 ] , and traffic analysis [ 4 ] [ 5 ] . These advanced algorithms enable more efficient networking management and resource allocation, contributing to improved networking reliability and user experience. However, traditional deep learning models present significant challenges in this domain. The adaptation of these models to different tasks often necessitates high engineering costs, as each new application may require bespoke model architectures and extensive retraining. Furthermore, these models frequently exhibit suboptimal performance when deployed in environments with unseen data distributions, thereby limiting their robustness and generalizability [ 6 ] .These limitations impede the scalability and practicality of implementing deep learning solutions across diverse and dynamic network environments. To address these limitations, recent research has explored the application of Transformer-based Large Language Models (LLMs) in networking, giving rise to LLMs for Networks (LLMs4Net) [ 7 ] . These approaches leverage the pre-trained, network-specific knowledge of LLMs to solve multiple network problems with a single model, significantly reducing engineering costs and demonstrating promising performance. Nevertheless, existing LLMs4Net approaches are constrained by two primary limitations: • Excessive Computational Overhead. The Transformer backbone of LLMs operates with quadratic time complexity, making it unsuitable for real-world network applications [ 8 ] where data flows are large and continuous. This high computational demand poses significant challenges for deployment on resource-constrained network devices such as routers and AR headsets, which lack the necessary processing power to handle such intensive workloads. • Significant Redundant Storage Consumption. LLMs encompass a vast array of knowledge beyond networking, leading to substantial memory usage in the form of numerous parameters [ 9 ] . This extraneous information not only fails to contribute to networking-specific tasks but also escalates the operational costs associated with deploying these models in real-world scenarios. To overcome the limitations identified in prior LLMs4Net approaches, this study proposes a novel framework for extracting domain-specific knowledge tailored to networking applications from LLMs. By focusing on networking-relevant information, this approach mitigates the influence of extraneous knowledge, resulting in a more compact model size. Furthermore, it significantly reduces time complexity, thereby lowering both training and inference overheads. Consequently, this method enhances the cost-effectiveness of deploying large language models within the networking domain. Inspired by the impressive breakthrough of Deepseek-R1 model [ 10 ] , which employed knowledge distillation to achieve impressive performance and low training cost, we develop Mamba4Net . Mamba4Net is a framework that distills the networking-specific knowledge of large Transformer-based LLMs into a more compact hybrid Mamba-based model. This approach enables the compression of the model’s scale, leading to a reduction in memory usage and an enhancement in computational efficiency. Specifically, it transforms the complexity from quadratic to linear, thereby minimizing computational overhead. However, previous knowledge distillation approaches suffer from unstable and inefficient training processes due to random weight initialization. To tackle these challenges and achieve the extraction of networking-specific knowledge and effective cross-architecture distillation, we make the following contributions: 1. Domain Knowledge-Oriented Cross-Heterogeneous Distillation (DKO): A novel method that distills networking-specific knowledge from Transformer LLMs to Mamba models, reducing quadratic complexity to linear while filtering out irrelevant information. 2. Cross-Heterogeneous Weight Reusing (denoted as CWR): We propose a creative weights initialization method, namely cross-heterogeneous weight reusing, which has been developed to minimize the instability and computational overhead during the cross-heterogeneous distillation phase. This approach differs from the traditional method of randomly initializing the student model by transforming the teacher’s attention block weights (from Transformer-based LLMs) into compact factors by decomposing high-dimensional matrices into low-rank subspaces. These compact factors are then used to initialize the Mamba-based student’s parameters, effectively narrowing the representation gap and facilitating more effective knowledge transfer. By focusing on the essential components of the teacher model’s weights, CWR avoids unnecessary overhead and provides a robust, architecture-aware method for achieving efficient distillation across disparate neural network designs. 3. Comprehensive Experiments and Open-Sourced Code: We conducted a series of experiments to validate the performance and cost of the proposed Mamba4Net , which has been employed in three distinct networking tasks, including viewport prediction, adaptive bitrate streaming, and cluster job scheduling (the same as for NetLLM). The experiments results demonstrate consistent improvements in both performance and computational efficiency, particularly in resource-constrained scenarios where resources are limited. The experimental code is available on GitHub, ensuring reproducibility and enabling further exploration by the research community. Our work take three networking tasks at use cases. The remainder of this paper is organized as follows: Section II offers an overview of the background and a survey of related work. Section III elaborates on the methodology underlying Mamba4Net . Section IV details the experimental setups and presents the experimental results. Sections V and VI discuss and conclude the study, respectively. II Background II-A Large Language Models for Networking Recent researches leverage LLMs4Net [ 7 ] can be segregated into three primary paradigms: zero-shot adaptation, fine-tuning adaptation, and reinforce-learning (RL) integration. Zero-shot adaptation uses LLMs’ general-purpose knowledge directly, without additional training on task-specific data. This approach is widely applied in networking tasks such as anomaly detection [ 11 ] , quality of service (QoS) analysis [ 12 ] , and configuration management [ 13 ] . For example, Sun et al. employed LLMs as aligners in UAV cluster networks to comprehend node status and issue high-level control directives [ 14 ] . While this paradigm eliminates the overhead of specialized training, it may struggle to achieve the accuracy required in complex or highly specialized tasks. In contrast,

Read the complete text