TransMamba: Fast Universal Architecture Adaption from Transformers to Mamba
Xiuwei Chen, Sihao Lin, Xiao Dong, Zisheng Chen, Meng Cao, Jianhua Han, Hang Xu, Xiaodan Liang · 2025
Read TransMamba: Fast Universal Architecture Adaption from Transformers to Mamba by Xiuwei Chen, Sihao Lin, Xiao Dong, Zisheng Chen, Meng Cao, Jianhua Han, Hang Xu, Xiaodan Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15130.txt.
Introduction
Xiuwei Chen and co-authors transfer knowledge from Transformer models into the newer Mamba architecture, published in 2025. Transformers power many leading foundation models, and a rich stock of pretrained versions exists. Mamba models offer gains in speed, since their cost grows more slowly with input length, but training them from scratch is costly.
Their two-stage method, TransMamba, first uses a pretrained Transformer to set key parts of a Mamba model. Special cloning and layer schemes bridge gaps in shape and size. A second stage then distills knowledge with layer-wise scaling that adapts as training goes. The aim is fast adaptation that works for both single-mode and multi-mode tasks, reusing what Transformers already learned.
Catalog record
| CREATOR | Xiuwei Chen, Sihao Lin, Xiao Dong, Zisheng Chen, Meng Cao, Jianhua Han, Hang Xu, Xiaodan Liang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-15130
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TransMamba: Fast Universal Architecture Adaption from Transformers to Mamba Xiuwei Chen, Sihao Lin, Xiao Dong, Zisheng Chen, Meng Cao, Jianhua Han, Hang Xu, Xiaodan Liang Abstract Transformers have been favored in both uni-modal and multi-modal foundation models for their flexible scalability in attention modules. Consequently, a number of pre-trained Transformer models, e.g., LLaVA, CLIP, and DEIT, are publicly available. Recent research has introduced subquadratic architectures like Mamba, which enables global awareness with linear complexity. Nevertheless, training specialized subquadratic architectures from scratch for certain tasks is both resource-intensive and time-consuming. As a motivator, we explore cross-architecture training to transfer the ready knowledge in existing Transformer models to alternative architecture Mamba, termed TransMamba. Our approach employs a two-stage strategy to expedite training new Mamba models, ensuring effectiveness in across uni-modal and cross-modal tasks. Concerning architecture disparities, we project the intermediate features into an aligned latent space before transferring knowledge. On top of that, a Weight Subcloning and Adaptive Bidirectional distillation method (WSAB) is introduced for knowledge transfer without limitations on varying layer counts. For cross-modal learning, we propose a cross-Mamba module that integrates language awareness into Mamba’s visual features, enhancing the cross-modal interaction capabilities of Mamba architecture. Despite using less than 75 % 75\% of the training data typically required for training from scratch, TransMamba boasts substantially stronger performance across various network architectures and downstream tasks, including image classification, visual question answering, and text-video retrieval. The code will be publicly available. 1 Introduction Transformer ( Vaswani et al. 2017 ) architectures have had a profound impact on the computer vision community ( Dong et al. 2022 ; Dosovitskiy et al. 2021 ; He et al. 2016 ; Liu et al. 2021 ; Simonyan and Zisserman 2015 ) , with their flexible scalability of attention modules considered crucial to their success. Despite their popularity, Transformers encounter obstacles due to the quadratic computational complexity of their attention mechanism ( Brown et al. 2020 ) , leading to increased computational expenses and memory usage. Consequently, this results in significant challenges in model optimization and extension, hindering their widespread applications. In response to this challenge, recent research introduces the subquadratic architectures, such as Mamba ( Gu and Dao 2023 ; Fu et al. 2023b ; Gu et al. 2023 ; Liu et al. 2024b ; Yang et al. 2024 ; Zhu et al. 2024 ; Qiao et al. 2024 ; Li et al. 2024 ) , RWKV ( Peng et al. 2023 ; Duan et al. 2024 ) . However, training specialized subquadratic models from scratch for diverse downstream tasks poses a significant computational burden, as shown in Figure 1 , with higher carbon dioxide emissions. Fortunately, we observe various Transformer-based pre-trained models, such as LLaVA ( Radford et al. 2021a ) , CLIP ( Radford et al. 2021a ) and ,, are publicly available. A natural question arises: Can we transfer the knowledge from pre-trained Transformer-based models to subquadratic models, such as Mamba? In this paper, we aim to explore the feasibility of transferring knowledge from these widely available Transformer-based pre-trained models to the subquadratic models, such as Mamba, for more cost-effective and efficient training. Figure 1 : Comparisons of CO_{2} emissions and training cost among different methods. Compared to Mamba, TransMamba uses less data and requires shorter training time. Specifically, we address two critical challenges in our research: 1) Tackling Cross-Architecture Learning : This involves adapting knowledge from one architecture framework (pre-trained Transformer models) to another (Mamba models). Besides, we must ensure that the transfered knowledge retains its effectiveness and enhances the performance of the target model without compromising its structural integrity. 2) Equipping SSM-Based Models with Cross-Modal Interaction Capabilities : This involves developing methods to seamlessly integrate and process information from different modalities, such as text and images, to enhance the versatility and application of SSM-based models in complex tasks. We should ensure that SSM-based models can understand and leverage the relationships between various types of data. Regarding the first challenge, our approach introduce a two-stage strategy to leverage the knowledge distilled from pretrained Transformer models to enhance the training efficiency and performance of SSM-based models. We first use a simple MLP layer to transfer the mismatched representions into an aligned latent space. Therefore, cross-architecture distillation at intermediate layers becomes achievable. However, there is an inconsistency correlation between the last Transformer layer and different-level SSM-based layers. We use the cosine similarity to evaluate the correlation between layers. And then, adaptive knowledge distillation is proposed to transfer knowledge, with higher similarity features receiving lower weight coefficients, and vice versa. To address the inconsistent optimization issues in the dual paths of the bidirectional Mamba for processing image information, we propose a bidirectional (Forward/Backward) distillation strategy to optimize different processes separately. Relying solely on distillation may not effectively facilitate knowledge transfer, especially in large models. Weight reuse is crucial for ensuring the accuracy and efficiency of transferred knowledge. Model weights of similar architecture can be directly reused for initialization. However, cross-architecture weight reuse is challenging due to structural differences. Inspired by ( Samragh et al. 2023 ) , we initialize SSM-based models from the pretrained Transformer models. One difference between them is similar architecture and cross-architecture (Attention and SSM). For structural difference, we retrain the weight except for QKV projection layer. For dimension difference, partial important weights from the Transformer models are used to match the dimension of the SSM-based models. Weigth subcloning also accelerates the convergence. Moving on to the second challenge, the absence of cross-modal interaction capabilities ( Hu et al. 2024 ; Dat et al. 2024 ) in existing SSM-based models poses a significant limitation for various multi-modal applications. Inspired by cross-attention mechanisms ( Chen et al. 2023 ) , we propose a cross-Mamba module to integrate multi-modal understanding capabilities into SSM. By stacking the Mamba and cross-Mamba modules, the new Mamba model is empowered to effectively handle vision-language tasks while maintaining efficiency and performance. This work primarily selects Mambas as the implementation of SSM-based models. We use Two-phase to transfer knowldge from pretrained Transformers to Mamba and its variant. Our method is fast and universal as it can boost the training of Mamba and apply to various architectures. In addition, we validate the generalization of the proposed method with thorough experiment settings including image classification, video retrieval, and visual question answering. We claim the following contributions: \bullet Fast and Universal Framework : We propose a fast and universal two-stage framework that transfers the knowledge of existing pre-trained Transformer models to new SSM-based models, which enhances both training efficiency and subsequent performance at a low cost. \bullet Weight Subcloning and Adapative Bidirectional Distillation (WSAB) : We use the weight subcloning to the cross-architecture transfer that efficiently leverages the pre-trained knowledge. Additionally, we propose a adaptive Forward/Backward distillation method for mamba architecture and endow Mamba with multi-model interaction capabilities. \bullet Comprehensive Validation : We validate the proposed method across a wide range of backbone architectures and versatile applications, including visual question answering, video retrieval and image classification. 2 Related Work Transformers Transformers ( Dosovitskiy et al. 2021 ; Dong et al. 2022 ; Liu et al. 2021 ; Cao et al. 2021 ; Cao et al. 2023 ) have demonstrated powerful perceptual capabilities for