Mamba-3D as Masked Autoencoders for Accurate and Data-Efficient Analysis of Medical Ultrasound Videos
Jiaheng Zhou, Yanfeng Zhou, Wei Fang, Yuxing Tang, Le Lu, Ge Yang · 2025
Read Mamba-3D as Masked Autoencoders for Accurate and Data-Efficient Analysis of Medical Ultrasound Videos by Jiaheng Zhou, Yanfeng Zhou, Wei Fang, Yuxing Tang, Le Lu, Ge Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20258.txt.
Introduction
Six authors led by Jiaheng Zhou teach a lean video network to read medical ultrasound clips with few labelled cases. Ultrasound film is a core clinic tool, yet labelled sets are scarce and film is hard to mine. Their net, built on the Mamba line, keeps the 3D shape of film so space and time links stay clear. A ring of global tokens pools the wide scene in each pass.
Pre-training hides parts of clips and makes the net guess them, a self-taught warm-up. The hide plan chains space and time gaps to fit many clip kinds. Tests on four sets, across two scan tasks, put the net at the top of its class. Skill held up with few labels, a key draw for real wards.
Catalog record
| CREATOR | Jiaheng Zhou, Yanfeng Zhou, Wei Fang, Yuxing Tang, Le Lu, Ge Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20258
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mamba-3D as Masked Autoencoders for Accurate and Data-Efficient Analysis of Medical Ultrasound Videos Jiaheng Zhou, Yanfeng Zhou, Wei Fang, Yuxing Tang, Le Lu, Ge Yang Abstract Ultrasound videos are an important form of clinical imaging data, and deep learning-based automated analysis can improve diagnostic accuracy and clinical efficiency. However, the scarcity of labeled data and the inherent challenges of video analysis have impeded the advancement of related methods. In this work, we introduce E-ViM³ , a data-efficient Vi sion M amba network that preserves the 3D structure of video data, enhancing long-range dependencies and inductive biases to better model space-time correlations. With our design of E nclosure Global Tokens (EGT), the model captures and aggregates global features more effectively than competing methods. To further improve data efficiency, we employ masked video modeling for self-supervised pre-training, with the proposed Spatial-Temporal Chained (STC) masking strategy designed to adapt to various video scenarios. Experiments demonstrate that E-ViM³ performs as the state-of-the-art in two high-level semantic analysis tasks across four datasets of varying sizes: EchoNet-Dynamic, CAMUS, MICCAI-BUV, and WHBUS. Furthermore, our model achieves competitive performance with limited labels, highlighting its potential impact on real-world clinical applications. Our code is available at https://github.com/HenryZhou19/E-ViM3 . Figure 1 : Mamba-3D as masked autoencoders. By preserving the structure of video data, Mamba-3D provides a stronger inductive bias for masked video modeling than ViT [ 12 , 60 ] or vanilla Mamba for visual data [ 73 , 38 , 31 ] , both of which operate on flattened 1D sequences and rely heavily on positional encodings. This enables effective self-supervised learning with limited data. Figure 2 : The pipeline of self-supervised pre-training and fine-tuning of the proposed E-ViM³ model. The upper section represents the pre-training phase; the lower section represents the fine-tuning phase. Initially, the video is embedded as 3D patches but not directly flattened into a 1D sequence. During pre-training, the proposed Spatial-Temporal Chained masking is applied, and masked tokens are removed for efficiency. The 3D-structured tokens, along with the proposed Enclosure Global Tokens, are inputted into the Mamba-3D encoder. The pre-training task is to restore the masked patches using a decoder, while downstream tasks leverage the features extracted from the global tokens. Further details can be found in Section 3 . A complete architecture diagram is available in the supplementary material. 1 Introduction In medical imaging, ultrasound video is a widely used modality that requires real-time clinical interpretation, in which missed or incorrect diagnoses often occur due to various human factors, underscoring the importance of AI-assisted analysis [ 54 ] . However, the lack of labeled data has greatly hindered research progress in this field as it often leads to serious overfitting issues in practice. Pioneering works by Ouyang et al . [ 46 , 47 ] have established a well-labeled and relatively large dataset of cardiac ultrasound videos, making a significant contribution to related research. However, for many other types of medical ultrasound videos, the lack of high-quality public datasets remains a real challenge. Therefore, it is both important and practical to develop powerful and data-efficient models for analyzing medical ultrasound videos. Self-supervised pre-training has been proven to be effective in learning visual features without specific labels, demonstrating feasibility in various scenarios involving images [ 20 , 6 , 21 , 3 ] and videos [ 18 , 19 , 66 ] . Most approaches utilize either contrastive learning [ 10 , 9 , 48 ] or Transformer-based masked autoencoding [ 67 , 60 , 65 ] . However, the reliance of contrastive learning on negative samples makes it unsuitable for medical video applications, as there is a natural dominance of similarities between different samples. Consequently, this paper focuses on masked video modeling as the pretext task. Furthermore, Transformer-based models lack inherent inductive biases and heavily rely on large datasets to learn meaningful visual patterns [ 12 , 62 ] . Additionally, Transformers often struggle with long or high-resolution videos due to the quadratic complexity of their attention mechanism [ 64 , 1 , 17 ] , and existing improvements are made at the expense of representation quality and convergence efficiency [ 55 , 26 ] . These limitations can lead to inadequate training and increased hardware constraints, ultimately affecting performance. The recently emerged selective state space model, commonly known as Mamba [ 11 , 15 ] , offers a promising solution to these challenges. Mamba employs latent states and the selective scanning mechanism to model relationships between input and output sequences, providing a more explicit inductive bias for sequential data than Transformers. In addition, with linear complexity and hardware-efficient implementation, Mamba is well-suited for handling long sequences. However, Mamba blocks use 1D scanning instead of attention, making the positioning of global tokens particularly important, especially when handling long sequences derived from 3D data. And effectively integrating Mamba’s 1D scanning with masked video modeling of 3D data remains an open challenge. In this work, we present E-ViM³, a Mamba-3D network with Enclosure Global Tokens (EGT) for data-efficient analysis of medical ultrasound videos. By employing the proposed Spatial-Temporal Chained (STC) masked video modeling as an effective pretext task, we pre-train the E-ViM³ model and fine-tune it entirely on high-level semantic downstream tasks. Experimental results demonstrate that our model, combined with this training strategy, surpasses previous state-of-the-art methods in left ventricular ejection fraction (EF) prediction and breast cancer classification across four public datasets: EchoNet-Dynamic, CAMUS, MICCAI-BUV, and WHBUS. Furthermore, our method is label-efficient, as it enables the analysis of medical ultrasound videos even with extremely limited labeled data. Our main contributions are as follows: • We introduce E-ViM³, a new Mamba-3D network incorporating Enclosure Global Tokens (EGT) that effectively aggregates global features on medical ultrasound videos, while maintaining the ability to leverage correlations in both time and space. • By caching index transformations between sequences after removing the masked tokens, we provide an efficient way to pre-train Mamba-3D blocks as a masked autoencoder. • We propose Spatial-Temporal Chained (STC) masking, a customized and easy-to-implement masking strategy that enhances the effectiveness of self-supervised pre-training for different medical ultrasound datasets. • Fine-tuning with limited labeled data from EchoNet-Dynamic validates that E-ViM³ is robust under label-scarce conditions. Moreover, results on small datasets such as CAMUS and MICCAI-BUV demonstrate strong generalization and data efficiency over prior methods. 2 Related Works 2.1 Analysis of Medical Ultrasound Videos In echocardiography, ejection fraction (EF) is crucial for diagnosing heart diseases. Studies typically segment the end-diastole (ED) and end-systole (ES) phases for EF calculation [ 69 , 68 , 29 , 37 , 58 ] , relying on additional label information. Recent efforts have been focusing on developing end-to-end EF regression models for practical clinical applications. Ouyang et al . [ 46 , 47 ] introduced the EchoNet-Dynamic dataset and a baseline method using R(2+1)D [ 63 ] for EF prediction, but their best results relied on an extra segmentation network. Reynaud et al . [ 52 ] treated each frame as a single token in the Transformer for arbitrary lengths. Mokhtari et al . [ 44 ] further experimented with a multi-level Transformer for multiple video aggregation of the same patient, achieving improvements at the cost of generality. Muhtaseb et al . [ 45