MedSora : Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generation
Zhenbin Wang, Lei Zhang, Lituan Wang, Minjuan Zhu, Zhenwei ZhangMachine Intelligence LaboratoryCollege of Computer science, Sichuan UniversityChengdu, Sichuan Province, China · 2024
Read MedSora : Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generation by Zhenbin Wang, Lei Zhang, Lituan Wang, Minjuan Zhu, Zhenwei ZhangMachine Intelligence LaboratoryCollege of Computer science, Sichuan UniversityChengdu, Sichuan Province, China (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01647.txt.
Description
Abstract Medical video generation models are expected to have a profound impact on the healthcare industry, including but not limited to medical education and training, surgical planning, and simulation. Current video diffusion models typically build on image diffusion architecture by incorporating temporal operations (such as 3D convolution and temporal attention).
Catalog record
| CREATOR | Zhenbin Wang, Lei Zhang, Lituan Wang, Minjuan Zhu, Zhenwei ZhangMachine Intelligence LaboratoryCollege of Computer science, Sichuan UniversityChengdu, Sichuan Province, China |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01647
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MedSora : Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generation Zhenbin Wang, Lei Zhang, Lituan Wang, Minjuan Zhu, Zhenwei ZhangMachine Intelligence LaboratoryCollege of Computer science, Sichuan UniversityChengdu, Sichuan Province, China Abstract Medical video generation models are expected to have a profound impact on the healthcare industry, including but not limited to medical education and training, surgical planning, and simulation. Current video diffusion models typically build on image diffusion architecture by incorporating temporal operations (such as 3D convolution and temporal attention). Although this approach is effective, its oversimplification limits spatio-temporal performance and consumes substantial computational resources. To counter this, we propose Med ical S imulation Vide o Gene ra tor ( MedSora ), which incorporates three key elements: i) a video diffusion framework integrates the advantages of attention and Mamba, balancing low computational load with high-quality video generation, ii) an optical flow representation alignment method that implicitly enhances attention to inter-frame pixels, and iii) a video variational autoencoder (VAE) with frequency compensation addresses the information loss of medical features that occurs when transforming pixel space into latent features and then back to pixel frames. Extensive experiments and applications demonstrate that MedSora exhibits superior visual quality in generating medical videos, outperforming the most advanced baseline methods. Further results and code are available at https://wongzbb.github.io/MedSora/ . 1 Introduction With the development and integration of technologies such as diffusion ( Ho et al., 2020 ; Sohl-Dickstein et al., 2015 ) , multimodal ( Radford et al., 2021 ; Ramesh et al., 2021 ) , pre-trained models ( Brown et al., 2020 ) , and model fine-tuning ( Hu et al., 2021 ; Li & Liang, 2021 ) , artificial intelligence generated content (AIGC) has achieved remarkable progress, sparking widespread interest in interdisciplinary medical fields. Ranging from image reconstruction ( Liu et al., 2023 ) and translation ( Wang et al., 2024 ) to the generation of virtual cases and the creation of simulation data ( Özbey et al., 2023 ) , AIGC underpin numerous innovations. This has significantly advanced the frontiers of computer-assisted diagnosis and precision medicine. Recently, advances in video generation models ( Blattmann et al., 2023 ; An et al., 2023 ; Huang et al., 2024 ) have propelled AIGC to new heights. However, due to the considerable complexity and significant resource demands of clinical video generation, research in this area remains in its nascent stages. In this context, our work aims to explore whether it is possible to create temporally coherent and realistic clinical medical videos. Significant progress has been made in generating realistic medical images using generative adversarial networks (GANs) ( Goodfellow et al., 2014 ) and diffusion models ( Rombach et al., 2022 ) . However, video is composed of pixel arrays spanning both temporal and spatial dimensions, necessitating meticulous attention to fine temporal dynamics and the maintenance of temporal consistency across frames. Consequently, generating stable, high-quality videos is a non-trivial task. To achieve this, most recent works in computer vision attempt to extend the spatial self-attention mechanism of images to spatio-temporal self-attention ( Khachatryan et al., 2023 ; Qi et al., 2023 ) , using pseudo-3D ( Singer et al., 2022 ) or serial 2D+1D convolutions ( Xing et al., 2024a ) to apply image diffusion models to video generation. To the best of our knowledge, the sole existing study in the medical field on video generation employs the spatio-temporal self-attention ( Li et al., 2024 ) . Those strategies integrate features from patches across different video frames via an expanded attention or convolution module, as illustrated in Figure 1 . Figure 1: Illustration of spatial attention, pseudo-3D or 2D+1D convolution, spatio-temporal attention, and our spatio-temporal Mamba. \mathbf{x}^{k} represents the feature map of the k -th video frame. Patches marked with ’*’ are computed together with the colored patches to aggregate their features. Despite their effectiveness in capturing contextual information across space and time, these methods have significant drawbacks: i) spatio-temporal self-attention requires each patch attends to all other patches in the video, which not only poses the risk of misdirecting the attention mechanism with patches that are irrelevant to video generation ( Liu et al., 2022 ) , but also significantly increases the computational complexity, presenting significant challenges in terms of resource allocation and maintaining cross-frame coherence, and ii) convolutional methods focus only on local areas and cannot capture the global receptive field, thereby reducing model performance. To mitigate these challenges, some recent works have incorporated Mamba ( Gu & Dao, 2023 ; Dao & Gu, 2024 ) into video generation, they typically replace the original convolution or attention modules of GANs or diffusion models with Mamba ( Gao et al., 2024 ; Mo & Tian, 2024 ) , or merely alter the scanning order of patches ( Park et al., 2024 ) without thorough exploration. These Mamba-based video generation models exhibit two primary limitations: i) existence of pixel-level bias. Recent advances in diffusion models for image generation illustrate that Mamba underperforms relative to certain Transformer in generating pixel-level data ( Wang et al., 2024 ) . And ii) absence of explicit spatio-temporal modeling. These models either maintain the same structural design of image generation models ( Mo & Tian, 2024 ; Gao et al., 2024 ) or simplistically modify the sequence of video frames ( Park et al., 2024 ) , assuming it to suffice for spatio-temporal modeling. We aim to address these challenges by advancing our approach with an optimized design of spatio-temporal Mamba and attention blocks. Spatio-temporal self-attention frameworks typically model the content within each video frame before conducting temporal modeling of patches at corresponding positions across frames ( Diba et al., 2023 ) . However, this approach has a significant limitation as it presupposes linear motion of patches along the temporal axis. This assumption leads to inadequate mutual attention among patches following non-linear trajectories across frames. To overcome existing drawbacks, we propose a supervised, optical flow representation alignment method seamlessly integrated into the video diffusion model. By using optical flow ( Fleet & Weiss, 2006 ; Shi et al., 2023 ) to implicitly control key features, our method addresses the visual consistency problems identified in previous work. An primary benefit is the facilitation of precise information transfer between frames through optical flow guidance, stabilizing the generated visual content throughout the video. Specifically, we initially employ a pre-trained optical flow estimation model ( Teed & Deng, 2020 ) to determine the optical flow of the source video. Following this, a self-supervised foundation model ( Caron et al., 2021 ) is utilized to extract key features of the optical flow from various perspectives. These key features are then sampled and leveraged to supervise the training of video diffusion model, without the addition of any extra training parameters. Video generation models process high-dimensional videos as cubic arrays, necessitating substantial memory and computational resources, particularly for high-resolution and extended-duration videos. To reduce the computational load, numerous video diffusion models ( Blattmann et al., 2023 ; An et al., 2023 ) are not trained on raw pixels, but first employ an autoencoder ( Kingma & Welling, 2013 ) to project video frames into low-dimensional latent space and then model this latent distribution. However, the pre-train