Mamba base PKD for efficient knowledge compression
José Medina, Amnir Hadachi, Paul Honeine, Abdelaziz Bensrhair · 2025
Read Mamba base PKD for efficient knowledge compression by José Medina, Amnir Hadachi, Paul Honeine, Abdelaziz Bensrhair (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01727.txt.
Introduction
José Medina, Amnir Hadachi, Paul Honeine, and Abdelaziz Bensrhair shrink image models with a lean network design in a 2025 paper. Big neural networks are hard to run on small devices. Their method pairs Mamba blocks with staged knowledge distillation. A large teacher passes skill to a chain of smaller students.
Each student uses selective state-space blocks to cut compute. Tests ran on two classic image sets, MNIST and CIFAR-10. On MNIST, a group of students kept 63 percent of teacher cost. They still matched the teacher at 98 percent accuracy. The weakest student used just 1 percent of teacher cost. It held 72 percent accuracy on that tiny budget.
Catalog record
| CREATOR | José Medina, Amnir Hadachi, Paul Honeine, Abdelaziz Bensrhair |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01727
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mamba base PKD for efficient knowledge compression José Medina, Amnir Hadachi, Paul Honeine, Abdelaziz Bensrhair Abstract Deep neural networks (DNNs) have remarkably succeeded in various image processing tasks. However, their large size and computational complexity present significant challenges for deploying them in resource-constrained environments. This paper presents an innovative approach for integrating Mamba Architecture within a Progressive Knowledge Distillation (PKD) process to address the challenge of reducing model complexity while maintaining accuracy in image classification tasks. The proposed framework distills a large teacher model into progressively smaller student models, designed using Mamba blocks. Each student model is trained using Selective-State-Space Models (S-SSM) within the Mamba blocks, focusing on important input aspects while reducing computational complexity. The work’s preliminary experiments use MNIST and CIFAR-10 as datasets to demonstrate the effectiveness of this approach. For MNIST, the teacher model achieves 98% accuracy. A set of seven student models as a group retained 63% of the teacher’s FLOPs, approximating the teacher’s performance with 98% accuracy. The weak student used only 1% of the teacher’s FLOPs and maintained 72% accuracy. Similarly, for CIFAR-10, the students achieved 1% less accuracy compared to the teacher, with the small student retaining 5% of the teacher’s FLOPs to achieve 50% accuracy. These results confirm the flexibility and scalability of Mamba Architecture, which can be integrated into PKD, succeeding in the process of finding students as weak learners. The framework provides a solution for deploying complex neural networks in real-time applications with a reduction in computational cost. 1 ITS Lab, Institute of Computer Science, University of Tartu, Estonia 2 Litis, Université de Rouen, France 3 Litis, INSA de Rouen, France {joseluis,hadachi}@ut.ee , [email protected] , [email protected] Keywords: Knowledge Distillation, Mamba Architecture, Image Classification, Machine Learning, Computer Vision. 1 Introduction A preliminary version of this work was presented as a short poster titled “Mamba-PKD: A Framework for Efficient and Scalable Model Compression in Image Classification” at The 40th ACM/SIGAPP Symposium on Applied Computing (SAC ’25) 1 1 1 A preliminary version of this work appeared as a conference poster: https://doi.org/10.1145/3672608.3707887 , This version extends the previous work by providing a detailed description of the methodology, additional experiments, and a more comprehensive discussion of results. Deep Neural Networks (DNNs) are large-scale models extensively used in natural language processing, image processing, and speech recognition tasks. These models are typically over parameterized to ensure generalization and effective feature extraction ( Gou et al., 2021 ) , but their size demands significant computational resources and generates challenges for real-time applications. Researchers have explored methods to address these issues by transferring the knowledge encapsulated in unwieldy models to more lightweight neural networks. The approach is known as Knowledge Distillation (KD) and involves compressing DNN architectures by transferring knowledge from a complex teacher model to a simpler student model without compromising accuracy and reliability ( Hinton, 2015 ) . The student model learns to mimic the teacher’s behavior by focusing on its outputs and neuron connections rather than raw data, reducing computational requirements while maintaining competitive performance ( Wang and Yoon, 2021 ) . KD encompasses the design of three components: knowledge modeling, distillation algorithms, and teacher-student architectures ( Gou et al., 2021 ) , as illustrated in Figure 1 . Extending the principles of KD, recent research has introduced Progressive Knowledge Distillation (PKD), which transforms the knowledge transfer process from a rigid student architecture to a progressive and dynamic one ( Dennis et al., 2023 ) . Instead of distilling knowledge all at once, PKD allows the student model to grow incrementally and refine its predictions over time by breaking down the teacher model into a series of smaller student models, or weak learners, that collaboratively approximate the teacher’s behavior. This progressive approach offers a flexible trade-off between inference cost and model accuracy, making it particularly beneficial for scenarios that require adaptive inference. Moreover, PKD can assemble the predictions of each student in parallel, enhancing coarse predictions by summing the contributions of different students without the need to reevaluate the entire model. This method not only improves efficiency but also adapts to varying resource constraints and performance requirements. Related to enhancing efficiency, new approaches based on State Space Models (SSMs) make significant contributions. These mathematical frameworks are ideal for modeling systems whose state evolves over time based on its previous states and current inputs, by considering simple matrix multiplications Gu et al. (2021) . SSMs serve as a foundation for capturing temporal dependencies, in tasks such as pixel sequences and time series in computer vision ( Nguyen et al., 2022 ; Lei et al., 2024 ) . However, traditional SSMs represent the entire system, treating all incoming information equally important, making them effective for modeling data but impractical for large sequences. Gu and Dao (2023) handled this inefficiency by introducing innovations in an architecture called Mamba. The central concept of this architecture is a Selective-State Space Model (S-SSM), which process information based on the current input, focusing on relevant information while discarding irrelevant data. This selective process reduces unnecessary computation, ensuring that resources are allocated to meaningful tasks. Figure 1 : Teacher-student framework for knowledge distillation, where knowledge can be transferred based on teacher-student outputs, features, and pure relations. The present work introduces a novel approach to model compression by integrating Mamba architecture within a PKD framework. This progressive distillation process enables the model to scale flexibly according to available resources and specific application needs. Mamba architecture significantly enhances efficiency by leveraging S-SSM, which focuses only on relevant inputs while discarding irrelevant data. This selective processing allows Mamba to act as an efficient backbone for progressively distilled models; for instance, in our results for MNIST, a group of seven student models trained progressively retained only 63% of the teacher’s FLOPs while maintaining the same teacher accuracy of 98%. An important feature of this framework is also the hardware-aware Mamba algorithm, which can manage several student models in parallel as they progressively refine their coarse predictions. By distributing computations across students, the overall inference time is reduced to that of the most complex student, plus a minimal aggregation step, which is much faster than the inference time required by the teacher model alone. In CIFAR-10, for example, our framework achieves 50% accuracy from teachers using only 5% of the teachers’ FLOPs in one student. This makes the framework particularly suitable for real-time applications and resource-constrained environments, where balancing efficiency and accuracy is crucial. The primary contributions of this work are: • Proposed Training Pipeline: A pipeline combining PKD with Mamba architecture to progressively construct and train student models from a teacher, ensuring incremental refinement. • Hardware-Aware Parallel Processing Algorithm: Efficient parallel management of student models, reducing inference time for complex data sequences and multiple students. • Flexible and Efficient Mamba Blocks: Student models are trained using Mamba blocks, which selectively handle important features while minimizing model size. This framework offers flexibility, facilitating the student’s training by adjusting levels of complexity and computational demands. • Validation on Benchmark Datasets: Experiments on MNIST and CIFAR-10 demonstrate the framework’s scalability and reduced computational cost. This paper is structured as follows: Section 2 provides an overview of works related to KD, PKD, and SSM. Section 3 delves into the theoretical foundation